Design

Onboarding Part 4 – Context

Jay ZipurskyLeave a comment

I’ve written about a couple of example cases and explained how onboarding bandaids often don’t work. Time to dive into the top of the marketing funnel.

Identify the Right Problem

I love challenging assumptions, so when I am told there is a problem and the cause is “that part”, I always take a step back and consider the system. As I wrote in part 2, many folks thought we had a problem with the social platform’s homepage since so many new users did not return. Since the homepage was something each of their experiences had in common, you can see how the conclusion makes sense.

However, we had little insight beyond web analytics. I did what I always do to understand what’s going on with underperforming products – I conducted user research in the form of a usability test. One testing approach would have focused on the homepage and I could have designed a test that only showed participants the homepage and evaluated how well it communicated the site’s purpose, or how easy it was to navigate, or determined new users’ initial actions, or other things. However, no one shows up at the homepage without signing up and no one signs up without first loading our landing page and they don’t get there without responding to an email or mailing from one of our partners.

All of that context is crucial to setting up the study to answer the question: Why are new users not engaging?

So, how do you communicate that context? You recruit representative users and have them experience all those steps themselves. When they reach the homepage, they will have the same knowledge and context as the people you hope to better understand.

The Problem

As I wrote earlier, it was quickly obvious that the initial messaging was confusing and misleading. The biggest problem was terminology which is not too surprising when you consider an audience from across the country with varying backgrounds.

Here’s just one example of many: The platform offered discussion forums – topic-based areas where members could post messages and comment on those messages. However, the initial email and landing page simply called these “groups”. Some participants thought these were membership-based clubs that met on a regular schedule and required a level of commitment and some even extrapolated from there to assume groups only met in-person.

This and other miscommunications affected participants differently. Some became interested in things the platform didn’t offer. Some became less engaged due to a misunderstanding. Others would have been more engaged had they properly understood. We even found that a strong emphasis on “making friends” had a surprisingly negative effect on many people we interviewed. So, viewing the homepage with these varied understandings was quite different than landing there without preconceptions. Assumptions and first impressions had been formed and they affected behavior.

Even though the stated goal for improving the homepage and onboarding was increasing engagement, I don’t think we could meaningfully do that without tackling this problem first. We really needed to reduce the number of variables we were designing for. Without upstream changes, we’d have to tackle these misunderstandings first in the product as part of onboarding.

This is why aligning marketing, business development, and product was the first priority. Even our small company had built silos that needed to be breached.

Fixing the Funnel

Our testing and research helped us better understand members’ priorities and the language they naturally used for the concepts we offered. Identifying the problem also gave us a wealth of knowledge we could use in designing the solution and, later, improving the homepage and the platform as a whole.

We shared our findings with the entire company, sharing test video to really prove the point. That got everyone on board to work on solving the problem.

With a clearly defined problem, design follows easily. 

Our design team worked collaboratively with other teams to iteratively redesign the marketing messaging and landing page. We improved and aligned terminology, we added adjectives to clearly communicate concepts, and we made sure to reflect the user priorities we heard in our research. We kept testing, too, and could see the improvement after each iteration. 

After the next major release, we found these design changes doubled the conversion rate.

It was a successful project and not at all what stakeholders thought they were asking for.

Thanks for reading. Stay tuned for the final part of the series where I discuss addressing core design problems affecting a released product.

Onboarding Part 3 – Pitfalls

Jay ZipurskyLeave a comment

This is part 3 of my blog series on onboarding and retention.

When new users struggle with a product, there are a few quick fixes teams often try. They are simple, cheap (sometimes), and have low product impact (there is little code needed). I’ll take a look at a few and point out some gotchas.

More Explanation

A common fix is to throw more information at the user.

The explanations come in many forms:

  • Additional documentation, maybe with links embedded in various parts of the application
  • Blocks of explanatory text embedded on the app pages (in-product documentation)
  • Marketing email campaigns
  • Outreach from customer support to help users one-on-one

These are pretty blunt instruments. They rely on your users reading or watching additional content. It’s a common refrain that “people don’t read” and that is often true. It’s especially true if the person is not motivated to do it. In trial or try out cases, the user is often looking to see how your product is going to help them, not read more about it.

In addition, these approaches have other drawbacks. In-product documentation engagement is hard to measure. Marketing emails are generally better at getting people back into the product than explaining concepts. Customer support is likely effective if they reach a user but is a very expensive and non-scalable solution.

Finally, in-product and standalone documentation are pretty static. If you’re not careful, you will end up with blocks of text that, once read, provide zero to negative value. Or, you will introduce a ton of documentation that few people read and must be maintained which, for a quickly evolving product, is not an easy task.

Tooltips, Product Tours, “In-App Experiences”

I am sure you’ve experienced the tooltip onboarding aid. Your first visit to an app pops up a number of small, helpful tips that point things out on-screen or guide you through one or more tasks. There’s no doubt these can be helpful and multiple companies have built tool businesses around this pattern.

The many 3rd party products are easy to integrate into applications and, better yet, enable anyone outside the engineering team to author and maintain the tooltips. However, authors need to take care that their tooltips and flows are well-tested and understand that every app update risks breaking them. The tooltip triggers and positioning often rely on HTML and CSS details that, if poorly understood, can result in surprising behavior. Poorly engineered front-end markup can also make for brittle tooltips.

Another weakness I’ve seen with these tools (though, perhaps someone has addressed this) is that it is very difficult to support guided flows across multiple page and state changes. For example, the tooltip logic might rely on a list containing only a single item and will only work the first time a user does the associated task. Or you may want tooltips to appear only if the application is in a certain state but that data is not available to the tooltip logic. These constraints force the author to only provide guidance in specific, limited contexts.

While each tool has its own technical strengths and weaknesses, the pitfalls to avoid are universal. Here are a few:

  • Obvious explanations – Try to avoid tooltips that explain obvious concepts or operations. Is it really necessary to point out the logout button with a note saying it logs you out? Maybe in some cases, but error on the side of brevity.
  • Excessive explanations – The social platform I described in part 2 had over 20 tooltips, displayed one at a time, that explained every button and section of the app’s home page. Torture. When we conducted usability testing, we found the homepage and main navigation was understandable and those tooltips were unnecessary.
  • Bad timing – Timing is everything. Someone learning a new application will simply not absorb information that is provided at the wrong time or in the wrong context. For example, if you need to explain how to schedule a new blog post, don’t explain it when I create a new, empty post (or worse, right after I’ve created my account!). A better time would be when I save a draft or attempt to publish a post.
  • Forced flows – Some guided flows take control away from the user, likely to avoid having the flow disrupted by an errant click or a bored user. Some force users down irrelevant (to them) paths. The social platform I worked on had at least a couple of distinct personas. If a new member, uninterested in social interactions, was forced down a guided tour about online chats and discussions, they may abandon the product. If the user is not engaged, they are unlikely to learn or remember these.
  • Poor explanations – Sometimes a product using unfamiliar jargon or concepts is a design deficiency you need to bandaid. If the author is unaware, the help can continue obfuscating things. Instead, treat the tooltips as an opportunity to help new users translate familiar concepts and terms into the application’s world. Be sure to write tips in the user’s language, even if that departs from marketing, branding, and product jargon.

This type of help is not a magic bullet. To be successful, you need to do the work to understand where and when your users need help and apply it sparingly.

As a matter of fact, the work needed to design in-product help is a lot like the work needed to design a product… Hmm… 🤔

Helping People

For new users to your product, your goal is to make them proficient users of the product. For many products, these people will need to learn something. Jared Spool has a nice analogy describing this learning destination as “target knowledge”. Your goal is to get them from their current knowledge to this target knowledge. Ideally, you can do this as an implicit part of the product design by doing your research and designing it in.

  • Know your users. Understand their starting knowledge, behaviors, and goals.
  • Design for new and experienced users. I will touch on this more in part 5 where refactoring the design to better support new users actually helped existing users as much or more.
  • Introduce concepts incrementally if you can. Provide help (in whatever form) when the user needs it.

You may still find areas where guides and other bandaids are needed but I bet you will be able to deploy them with precision and confidence.

Wrap Up

Like any product, feature, document, or anything with an audience, you need to do your research to understand how your users think, what their goals are, and what expectations and knowledge they come to you with. Without that work, even though these fixes don’t heavily impact the existing product, these aids will fail to help.

In parts 4 and 5, we’ll look at the cases I described in part 2. Both these examples illustrate cases where bandaids like these made little impact.

Onboarding Part 2 – Case Studies

Jay ZipurskyLeave a comment

In part 1, I described how retention issues affect businesses and how UX teams are sometimes asked to fix the problems.

It’s story time! Let’s take a look at a couple of real-life examples.

A Social Platform

The first product facing challenges is a social platform aimed at older adults. Think Facebook in terms of general functionality with additional virtual, live programming. Membership was paid for by a sponsoring organization, such as a health insurance company, and provided to their customers as a free benefit.

Prospective new members were contacted by the sponsor via email or snail mail brochures. This material contained a link to a landing page with more info where the older adult could go on to register as a new member.

The product suffered from low retention rates and there were a bunch of theories as to why, ranging from “it looks too complicated” to “there’s not enough activity” to “members can’t remember how to get back to the site.” In particular, there was a lot of focus on the site’s homepage, suspected of being overwhelming, confusing, or otherwise unusable. Since this was a member’s first and subsequent view of the platform, it made sense to ensure it was as welcoming and clear as possible.

But, was the homepage the problem? It is very challenging to get any feedback from disengaged members, let alone something as nebulous as first impressions from a website visit.

To better understand what a new user experienced, I designed a user research study that started at the beginning – with a simulated email from the participant’s health insurance company. During the session, we observed participants read the email, visit the landing page, registration page, and the homepage.

Even before we had completed the 5 planned sessions, it was obvious that the homepage was not the main issue. We discovered that the sponsor’s email and our landing page both painted an ambiguous picture. Participants simply did not understand what was offered. Many asked, “Is this a dating site?” (It is not.) “Is this a group that meets in my area?” (It is purely virtual.) Or, they interpreted ambiguous terms in their own ways. For example, we started gaining insight on the various interpretations of “meeting people” or even “learning” – terms we thought were straight forward.

This research was crucial for two reasons:

  1. It gave us valuable insight into how our audience interpreted our messaging, what terminology they used to describe things, and what resonated for them.
  2. It challenged existing assumptions and pointed to a problem no one knew we had.

Before we could tackle engagement, we needed to ensure new members were set up for success. Focusing on acquisition first would also help increase the odds that people who may be a good fit would sign up in the first place.

Software QA Test Tool

The second product was a test automation tool for software teams. It had a 2 week free trial and, like any trial, the hope was trial users could quickly get up to speed, discover the product’s value, and become paying customers.

When I joined this project, the product was mostly built and nearing public release. My initial usability testing with new users discovered a number of usability issues. However, due to the impending release, we could make only superficial changes.

After release, conversation rates and feedback from Sales and Support confirmed we had a problem. Product leadership tasked the team to “fix the onboarding.”

We tried different approaches that avoided big product changes:

  1. We put in a few subtle hint popups attempting to steer new users to the most relevant pages
  2. We broke up the very first interaction into a few steps to provide more explanation and force a couple of actions we thought were necessary to fully evaluate the product
  3. We added additional tooltip flows using Appcues to guide the new user through their first key tasks

While we saw limited success with each of these changes, they didn’t move the needle. Again, I continued usability testing to understand what trial users were doing and thinking. The results built on the initial testing. My initial conclusions clarified into an inescapable truth – the product had fundamental issues that no number of bandaids could fix.

New users were confronted with too many product features and concepts all at once. And, because of the design, there was no way to avoid or defer many of the most confusing or complicated aspects. As a result, many test participants failed to complete the most basic task a QA test tool should support – creating a test – during a test session. That is not a good first impression.

There was no way to “fix the onboarding” and we’d have to take a step back and figure out how to fix the product.

In part 3, I focus on the quick fixes we tried and their pitfalls. In parts 4 & 5, I’ll tell you more about how we tackled the problems described here.

Onboarding Part 1 – Retention Problems

Jay ZipurskyLeave a comment

This is part 1 of my series on user retention and onboarding.

Defining Terms

Let’s define a few terms.

Customer or user retention means keeping customers or users. What does it mean to “keep” a customer or a user? It means that a person keeps using your product or service over time. However, how you define and measure that depends on the business. I touch on some measurements next.

I may use “customer” and “user” interchangeably here. For many products, they are the same but for B2B products, they are different since a customer may easily have multiple users. The retention of a customer is clearly different than the retention of one of their employees who uses your product (though they are likely correlated).

Onboarding refers to the period starting when a customer or user begins using your product until the point they are able to get some value from it. For complex products, onboarding could be a multi-week process that includes training classes. For simple products, it could be instantaneous.

SaaS stands for Software as a Service and describes both technology and business model. SaaS products are delivered remotely over the internet (i.e., there’s always some component running in the cloud) and are sold on a subscription basis. I am primarily writing about this type of product.

Measuring Retention

How do you measure retention and what do user retention problems look like? Ultimately, they are revenue problems – the company isn’t meeting financial targets. This blog series isn’t about product management, sales, or marketing, so let’s skip a bunch of analysis and assume that the marketing funnel is delivering the right number of prospects, you’re charging the right price, and you think you have product-market fit… if only users would just stick around.

In short, you’re getting enough new users but they aren’t coming back for some reason.

Depending on your revenue model, you may not see a problem right away. For example, if you sell annual subscriptions, it could take a year to realize customers are not renewing at the expected rate. So, most companies identify metrics that may indicate retention problems well before the renewal comes around. Some common ones:

  • Active users
  • Frequency of key actions (e.g., logging in, running reports, executing automations, posting content)
  • Customer support calls
  • Response rate to outreach

Retention is such an important business metric that you can be sure someone outside of the user experience team will notice problems. When that happens, how might the UX team be called on to help?

Calling User Experience

UX is an obvious choice to bring in to fix retention problems. For a strong UX team, I would expect they tackle the problem like any other starting with research and followed by an iterative design process with lots of user feedback to solve it.

However, there are still many companies that don’t operate this way and attempt to fix the problem based on assumptions and quick fixes. In this case, the UX team may be prescribed work. There are a couple interesting contexts: low retention of paying customers and low retention of trial or freemium users. Let’s look at some of the things a designer may be asked to do. 

Losing Subscribers

Almost every product uses a subscription model these days. When retention of paying customers is a challenge, the UX team may be tasked with things like:

  • Improving the usability of key functionality because low usage means it’s too hard to use
  • Adding more help or documentation or pointers to these features
  • Designing more new features to fill an assumed gap

In these situations, it’s sometimes a little clearer to everyone what the problems are since it’s easier to talk to customers than to semi-anonymous people just trying out your product. These fixes also assume that paying customers have gotten through the onboarding stage successfully to some extent. If not, requested fixes will sound a lot like the ones below.

Free Trials or Freemium Products

Many companies offer free trials or a free tier hoping people will sign up and the product will sell itself. These are challenging models to get right and to diagnose the problems when things go wrong!

Trials have their own retention metrics and, as a bonus, you can get feedback on the order of days or weeks. When trial users leave and don’t come back after their initial session or two, UX may be asked to tackle things like:

  • Explaining the product’s value proposition when users first sign up
  • Walking new users through the product through wizards or tooltips
  • Doing more automatic setup or configuration
  • Adding video explainers because people just don’t read

I like to summarize these requests as fixing the onboarding.

In part 2, I take a step back and consider some real-life examples and see just how these issues cross organizational divisions and how deep they can go.

The User Onboarding Problem

Jay ZipurskyLeave a comment

“We have a user retention problem. They just aren’t coming back and it seems like new users don’t seem to know what to do. We need to improve our onboarding.”

I’ve heard this or something like it many times in my career.

The thinking goes like this: If users aren’t coming back then they must not be getting value. And if they aren’t getting value, that must mean that they couldn’t find the value. (We provide a lot of value!) If they can’t find it, then the onboarding is broken – it needs to better show them the value!

Quick Fixes

Chances are, if you’re experiencing retention problems, it’s a new product and your company is a startup. Startup culture is not afraid of issues or challenges! They eat them up for breakfast.

You’ve already determined that your retention problem is due to an onboarding problem, so you move fast and throw some things at the wall to see if they stick.

This image is a derivative work of Screenshot of Qt Creator 3.1.1
  • More explanatory text!
  • Less text!
  • Force the user to do the thing that shows value!
  • Tooltips!
  • Dedicated customer support managers!
  • More tooltips!

Some of your attempts bomb. Some of them seem to work for some people but not enough. Some of them work really well but won’t scale. Sometimes you stumble on a fix that works and mistake luck for knowing your user.

Does any of this sound familiar? This blog post series covers:

  1. What retention problems look like
  2. Common fixes and their pitfalls
  3. Digging down to the root causes
  4. Designing the solution
  5. Implementing the design

Spoiler alert: The problem and the solution involves understanding your users.

Stay tuned for part 1, coming soon.

Balancing Acts: Too Little vs. Too Much

Jay ZipurskyLeave a comment

pexels-photo-235990My previous post was about balancing problem definition against design and implementation. The balancing act doesn’t end there, though. Even after defining the problem, there are usually multiple solutions to choose from and each of those can be considered complete with varying levels of functionality.

The goal is to deliver a well-balanced solution that offers just the right amount of functionality. There are easily dozens of decisions about the design and implementation that can throw the balance off. I’ve seen and fallen into a few patterns where these design and implementation tradeoffs didn’t quite balance.

Too Much Left on the Punch List

Some teams (engineering, design, or product) push back on basic functionality to cut down on initial effort. For example, the design requires the user can create an object but, to save a few hours of work, we’ll defer implementing rename, delete, or edit until later. They argue that these are easy to add later and the engineering team can always make these changes manually, especially early on when there are few customers.

No matter your role, it’s tempting to accept this argument. As a product manager, you can fit in an additional feature. As a designer, you are freed up from considering some design details. As an engineer, less code means fewer tests and fewer bugs. (OK, maybe it just means different bugs. 😉 )

Don’t fall for the argument. Returning to “completed” features later after the product has grown is harder than it seems. Eventually, what would have been fairly quick to add before becomes difficult as the team loses familiarity with that area of the code or, worse, changes are made only to uncover surprising dependencies and new bugs (“What? That name could never change before!”).

Omitting such basic functionality, even if you expect people will rarely need it, can lead to your users:

  • feeling they have an unfinished product
  • frustrated by attempting to find something that’s not there
  • frustrated by the need to contact support
  • abandoning that functionality or the entire product

Teams often cut corners when they doubt that the functionality is here to stay. Perhaps the product vision is unclear or there has been a recent pivot that undercut trust in the vision. In that case, you either need to convince the team that this work is going to stick around or you need to do the researching, testing, or experimentation needed to convince everyone (including yourself) you’re on the right track.

Gold Plating

It’s easy to overcompensate in the other direction and build unnecessary or low-value functionality. Obviously, doing unnecessary work consumes development time better used elsewhere. Gold plating also risks:

  • complicating the user experience
  • additional documentation or training effort
  • creating a maintenance burden
  • additional customer support

Finding the Balance

casey-horner-460825.jpgThe performer finds balance through feedback, practice, and experience. Designers achieve balance the same way. The tightrope walker knows when they’ve fallen off the tight rope but it’s harder to know when a design is off balance.

Trying out design ideas with users is critical to get early feedback. As they use the design prototype to complete realistic tasks, are they using all the functionality? Are they missing functionality? What can be removed? What must be added?

As useful as usability testing is, it can’t replicate reality. It’s important to monitor how the design performs in real life. Luckily, we have a many tools at our disposal:

  • Usage analytics tools like Heap and Pendo
  • Customer support feedback
  • User research – interviews, observation, surveys, etc

How much time you put into collecting and analyzing data depends on the size and importance of the feature. However, in order to improve your experience, you need to do something. Learning from each balancing attempt will help you get the balance right earlier in the process and spend more time finding new problems to solve.

 

Balancing Acts: Thinking vs. Doing

Jay ZipurskyLeave a comment

barnum_and_bailey_grand_water_circusDoes your work ever feel like a circus? Yeah, mine too. I feel like I am always performing. My most popular performance is the balancing act – it turns out that product design is a series of balancing acts.

As designers and product managers, we are always balancing the good-cheap-fast triangle. No time to fully consider a design? Whiteboard it and check off “cheap” and “fast”. The lone UX designer with a bit of lead time? Check off “good” and “cheap”. Sometimes, the balance is determined by outside forces. Other times, we have more influence.

Let’s take a look at one of the most popular balancing acts that I call “Thinking vs. Doing”.

Thinking vs. Doing

How much time should you spend understanding the problem you need to solve and finding the best solution before starting implementation? You can ask this question when embarking on a new product design or when adding a small enhancement. Of course, the larger the project, the more important it is to understand the problem before getting too far with design. Larry Marine likes to say you should “avoid solving the wrong problem very well.” He’s absolutely correct.

Now, I use “thinking” facetiously. Understanding the problem requires more than thinking. It’s research and research is work. From the outside, though, it may look a lot like thinking, so you may be tempted to cut corners at this early stage. For example, I know that an idle engineering team can result in pressure to speed things up.

Here are some tips to do the right thing while keeping things moving and knowing when you are done.

Start with Assumptions

List your and your team’s assumptions about the product/feature as soon as you can. Involving the wider team will ensure you generate a comprehensive list. These assumptions include the market, users, proposed functionality, and technology. You can capture these as simple lists, provisional personas, requirements, business plans, or whatever combination works for you.

Next, for each assumption, determine your confidence level. Items with high confidence should still be validated, but these are also areas where you can press forward with design and implementation right from he start.

Involve the Entire Team

Is that idle team pressuring you? Get them involved in the research. At all times, it’s best to have engineers tag along on research activities and help analyze results. But, if the team is truly waiting on the research, you could quickly train a few people (who have the proper aptitude) to do some independent research activities.

Enough is Enough

Do enough research to validate or invalidate these assumptions and better define the problem space. The best guideline I’ve heard is to research until you are no longer surprised. As soon as commonalities start appearing, you should be able to firm up some of the initial assumptions and commit to bigger decisions.

You should also continue research activities throughout the life of the product (for example, regularly usability testing), so it’s OK to shift focus to solutions as soon as you can.

Solving the Problem

Once you think you’ve got the problem defined, it’s time to design the solution. This is another activity that needs careful balancing as you could easily search a long time before you find the “best” solution. At this point, I like to bias the effort towards finding a “good enough” design quickly.

Determining “good enough” is a blog post (or book) of its own. However, your problem definition will help determine what that means in the product’s context.

Keep Moving

It’s important for the entire team to continuously make progress. As a designer or researcher, one of your responsibilities is to communicate with the team. This communication needs to include more than deliverables – it needs to include the entire process so everyone shares a sense of progress.

Finally, after doing such a good job at balancing the problem discovery work, never lose sight of the problem you are solving. The act of design and feedback often leads to new insights into the problem and it’s important to incorporate those and adjust as you go.

My users! No, MY users!

Jay Zipursky1 Comment

One set of design challenges when working on enterprise products is the number of different people and roles involved. Many designers and product managers are familiar with the buyer vs. user split where the people using the product aren’t the same as those making the purchase decision.

Product teams have historically given more weight to the buyers and their opinions, requirements, etc. This is for good reason – if the buyer doesn’t buy, your product goes nowhere. Catering exclusively to the buyer usually results in a lousy product that sits on the shelf or is despised. With ever more SaaS products that are sold on a subscription basis, ignoring the end user will result in poor adoption and renewal rates.

Anyone practicing user-centered design knows the importance of studying and working with the people doing the actual work. So, why is it so hard to get access to these people?

While I was working on an enterprise file sharing product, I realized it wasn’t just a simple case of finding the right person to interview. Many of our contacts were IT staff either administrating or evaluating our product. I had many good meetings with these folks, but my frequent requests to meet with our product’s end users often went nowhere.

One day, an IT contact’s comment finally provided the insight to why I wasn’t getting past IT. IT departments are responsible for evaluating, purchasing, and maintaining their company’s systems. So, IT is also responsible for understanding their company’s employees’ needs. Asking to interview the users was either being interpreted as redundant or, worse, insulting!

Getting through these gatekeepers is challenging and requires persuasion and persistence. There are a few approaches to try:

  • Explain the importance of getting access to end users and how your research methods and first hand data are crucial to providing the best product for their money.
  • Treat the IT staff as surrogate users. Conduct interviews, usability tests, and other research and design activities  with them. However, push the boundaries of their knowledge and look for the opportunity to get access to actual users to answer outstanding questions.
  • Recruit representative users from outside the company. You’ll avoid possibly wasting time but may not discover needs and design solutions to help your actual customers.

Share your experiences in the comments.

Your data is not backed up

Jay Zipursky1 Comment

You think your data is backed up, but it’s not.

Okay, maybe it is right now, but will it be there when you need it? Will it be there when you realize you need it?

A while ago, realizing I was doing a poor job of maintaining a home computer backup, I signed up for an online backup service. Once the initial backup was complete, wow, did I feel better – I had one less stressor to nag me.

After several months of magical backup, something started to go wrong. Unfortunately, it wasn’t a disastrous, sudden crash. Unfortunate, because if I had lost everything one day, I would have simply restored the data and continued along just as one would expect to do with a backup. However, whatever it was sneaked up on me. I think the first sign I remember was iTunes losing track of certain files – apps or podcasts or music. I attributed it to other causes besides a drive failure… I had relocated the iTunes library to that drive and figured iTunes was simply confused. I ignored the continuing glitches for a while until I realized I was missing photos.

Well, no problem, I thought. I logged into the backup service, selected the folders I wanted restored, and confidently sat back and waited for the two large .zip files to download over my pitiful DSL connection. Problem solved, emergency dealt with, I set the work of actually restoring the files on the back burner. Life is busy, and I’d deal with this later.

After a week or two, I got back to the mundane task of restoring the files. I had requested two sets of backed up files but, searching my computer, could not find the second .zip file. I checked the backup site for the file, but it had expired and was no longer available. That sinking feeling began it’s downward journey. I attempted to restore the file set again, but the files were not listed. “Oh shit,” I thought.

In user experience, we speak about the mental models people build in their heads to explain how the world works. Sometimes, in the case simple physical systems for example, these models are accurate. For more complicated systems, such as the computer or device on which you’re reading this post, the model may be abstract but still accurate enough to explain cause and effect and predict how things will work. Other times, the model is just wrong, and that leads to mistakes, which can multiply as we misunderstand why the system isn’t working as expected.

Our mental models are constructed from our past experiences, experience with the system we are modeling (the “learning curve” could be called “mental model construction”), and other indirect sources of information such as documentations, friends’ or colleagues’ advice, etc. Until this point of the story, my mental model for online backups was based on my experience with my local backups. Once you backed something up, it was backed up until I decided to delete or overwrite the backup file. Pretty simple. My backup service even offered “unlimited data backup”, further reinforcing my mental model. If it was “unlimited”, then the backup file would never be deleted. Right? Wrong.

My “oh shit” moment was the realization that my mental model was completely wrong. Not only was it wrong but, man, was I an idiot for having constructed it. Me! A software engineer for many years should have known how this stuff works.

These backup services do not store unlimited data forever. They store an unlimited amount of data and keep “previous” file versions for 4 weeks. That means, if you delete a file, you have 4 weeks to restore it before it’s gone forever. That means, if you remove an external drive from the backed up computer the backup service will assume deletion and dump the files after 4 weeks (as a friend discovered). This also applies to files that disappear silently due to a slowly failing hard drive. Your data is not protected from such non-disastrous failures or from your own procrastination.

Is this a surprise to you? Review your backup provider’s web site and tell me if your mental model is correct.

My story does have a happy ending. I contacted the backup service’s tech support and they were able to recover the .zip file since it had been scheduled for deletion but not actually deleted. That was pure luck and I’m happy to have learned my lesson without earning the scars.