Stylus Adventures

The iPad was designed for fingertips, plain and simple, but that hasn’t stopped countless companies from creating input devices of all shapes and sizes. Over time, they’ve gotten bigger and smaller, smarter and simpler, and this summer we sat down to see if we could take advantage of some of them to improve forScore’s annotation experience with version 8.0.

Of course, the majority of these products work right out of the box. They’re basically capacitive sponges on sticks, and that’s that. To really make a meaningful impact, though, they also need to communicate with the iPad to offer additional features like palm rejection, and the best way to do that is with Bluetooth Smart (aka Bluetooth LE, Bluetooth 4.0). That narrowed things down for us, and we identified three companies that offered something we thought could help our users: Adonit, FiftyThree, and Ten One Design.

Playing Ball

The first step was to try out each of these companies’ SDKs (software development kits). Unfortunately these libraries can be quite large and would have more than tripled the size of forScore without benefitting most users at all. Worse, still, is that some of them didn’t even work reliably and since SDKs can’t be viewed or modified, any bugs they introduced would be difficult to diagnose and fix.

Meanwhile, we’d already been working to implement direct connectivity with the iRig BlueBoard and had set up all of the infrastructure required to communicate not only with it, but with any Bluetooth Smart device (except for Ten One Design’s Pogo Connect, which requires authentication and couldn’t be used without their SDK). Best of all, it took up a mere hundredth of the size of these SDKs. We decided to strike out on our own and create an experience that was on par or better than what we could have offered by relying on third-party code.

We designed an interface to connect with devices. We created our own palm rejection techniques. We used pressure sensitivity to create a more natural drawing experience, even on devices that have the required hardware but were being artificially limited by their SDKs. We were proud of it, and it all worked well together. Sometimes.

Too fine a point

The best stylus is the one that enhances the iPad’s design rather than trying to work around it. The best example of this is the FiftyThree Pencil, but it’s an outlier in a market otherwise flooded with devices trying to replace pens rather than fingers.

While racing to create the tiniest capacitive tips possible, companies used elaborate tricks to try and strong-arm the iPad into reliably interpreting their device’s input. Once we released forScore 8.0 and began receiving feedback from our users, it became clear that we hadn’t accounted for everything. While the Pencil worked perfectly, the Jot Script was another matter. Depending on the angle of the stylus, the input would flail around by as much as twenty pixels—not a great experience. So we took another pass with forScore 8.1 and uncovered a terrible Rube Goldberg machine.

The root of the problem is this: iPad screens can’t “see” the tiny tips on these devices. So, to help, the devices themselves send a small amount of electricity back into the screen to strengthen their capacitance. Unfortunately, this produces a sort of blurring effect that travels in whichever direction the stylus is tilted, but only along one axis—left or right in portrait orientation, and up or down in landscape orientation. Because of this, we assume that the iPad’s screen has a natural weft that this process exposes. As you draw, the center of this blurry area is interpreted by the iPad as the touch point. The more you tilt, the further away from the actual tip that point moves.

Balancing act

To counteract this problem, some styli use accelerometers to try and figure out how tilted they are and account for these differences on the software side of things. Although these sensors provide a good idea of how tilted a device is, there’s no way to know which direction the stylus is tilted. If it’s tilted up or down, the bleeding effect is minimal, but if it’s tilted left or right it’s a big problem. So for this to help, the app also needs to know what your writing style is: do you write with the pen pointing horizontally, diagonally, or vertically? Of course, since you’re writing, the accelerometer’s data fluctuates wildly and the only way to get reasonable information is to average out the data coming from the stylus over time, which in turn makes adjustments feel sluggish.

But wait, it gets worse! The stylus measures tilt, but not in relation to the iPad itself. So if you’re drawing with the iPad flat on a table, things work out all right, but if you set your iPad up on a music stand, all of the numbers are off. To counteract this horrible complexity, these companies use the iPad’s accelerometer and adjust the numbers to get a sense of where their device actually is in relation to the iPad’s screen.

Does it work? In short, no. In long, still very much no. Using some of these devices with their company’s own apps, the experience is frustrating and wobbly. It’s no wonder, given the amount of contortion required to even get these things to function on a basic level. Heck, some of these companies don’t even try to account for this problem and just let your drawings sway all over the place—companies that are known for their expertise in digital drawing.

Centered

We tried every way we could think of to fix these problems, but there’s no amount of magic that can create a consistent, great experience with a fine-tipped stylus. In the end the best way to deal with these devices was also probably the easiest: calibration. Using your natural writing position to tap a target gives us the best sense of how much to adjust drawings so they’re in the right spot. Instead of trying to move things around as the sensors broadcast their data, we create an ideal but consistent adjustment that doesn’t leave you guessing where your lines will end up.

Using a stylus will never be as comfortable or accurate as using a pen or pencil, but we think the strides we’ve made in forScore 8.1 certainly help. If you’ve got your stylus already, rest assured that we’ll keep tweaking and thinking about how we can continue to extend and surpass the experiences these companies have defined. If you’re still deciding which stylus to get, we continue to strongly recommend the FiftyThree Pencil.

Disclaimer: We have no insider knowledge of how these devices work or what, if any, techniques these companies actually utilize in their SDKs. These educated guesses are speculative and our theories have held up to technical scrutiny, but they might be completely wrong. We also want to express our gratitude toward these companies for their willingness to try and make something great and new. Although we see some significant hurdles with certain implementations, we all want the same thing—a great drawing experience in a variety of forms so everyone can find one that suits them.