For a long time I’ve thought about building a personal digital assistant. This desire gets a little boost every new Iron Man movie, but recently I thought about it in a new light via blogger Sacha Chua. She was talking about building tools for your future self. One expects (or hopes!) to be around for a while yet, and technology is not going away. So how do you set up digital tools to support yourself now and into the future?
This post is a bit of a reflection on these ideas with no real conclusion. I’m just feeling out the idea.
Digital assistants
So by a personal digital assistant, I mean something like a lackey, a dogsbody, a personal assistant that’s… well… digital. It’s job is to take care of small, knowledge-based tasks like booking flights, tracking calendars, and recording and presenting data about myself (a la the Quantified Self movement) In particular, being digital it will have limited effect on the world. More a listener than a speaker or do-er.
In its most basic form this would be a set of tools for me to use to streamline my life. I use Emacs’ org-mode extensively for this purpose. It’s an easy way to collect my thoughts, store my calendar, and track tasks.
The more desirable ideal would be something with autonomy - the assistant would be able to glean information from the world, and perform required tasks on its own. To get to this stage it must:
- Have enough knowledge for making decisions
- Have enough agency to perform actions
- Have enough consistency to be trusted with the responsibility
To give it enough knowledge, you need to give it enough data, context and tools to analyze the data. To give it agency, it must have a variety of verbs available to it, which might be as simple as a text message or email, up to filling out Internet forms and interacting with lights or garage doors. And to have consistency, it needs to have reliable intentions and be able to have those intentions read.
Some of my tools already have elements of these requirements. I have configured org-mode at work to automatically track and markup my work in a way that’s useful later on. It’s trivial for me to get a list of all my work journal entries for the last quarter, filtered by topics. My bathroom scales automatically update a database whenever I make a measurement. But my tools are a little ad-hoc and often don’t talk to each other.
I also have an assistant app on my phone. I can trust it to take vocal instructions to set reminders or maybe query the web. Other things (like change the music) or book tickets for flights, I wouldn’t. So it’s pretty good, but not great. And I’m limited by what the developers include in the app.
The hypothetical
What would the ideal personal digital assistant look like?
As exciting as the idea of Jarvis is, I think what might be more realistic (ha!) or useful is a swarm of ambient intelligences. Or a single entity with different personas. Perhaps there’s a specific assistant output who tells me about tasks and my calendar. But then there might be a different one in the kitchen showing useful general information. And another one that tracks fitness and sends me reports.
I’m not precisely sure what the difference between a single assistant entity with different modes (like a personal assistant who briefs you on different parts of your life), versus several assistant entities (like this School of Comedy skit). I feel like for them to all interact, the backend should be one monolithic thing, but all the front-ends can be whatever they like.
I’d need to be able to access it from my desktop, my phone, offline and online. It’d be nice if a front-end was customized to the style of information and behaviour I’d want from that particular position. Common household data in my kitchen, creative projects in my office, and social or commercial things on my mobile or wearable device.
It’d be nice if it were hot-swappable so I could modify it on the fly without shutting the entire apparatus down.
Although my partner is not as tech-loving as me, it would be good if the system could handle multiple people, though that brings in many more complications of data storage, segregation and context-switching.
Implementation
My instinct would be to make it all myself. Maybe in Python for ease-of-use. But I tend to want to design whole architectures to completion, which is a process doomed to fail. As Sacha suggested, the better way would be to build up something small over time. Agile instead of Waterfall for the project managers amongst you.
Another approach is to borrow a system someone has already made and bend it to my will. Huginn seems ideal, but is written in Ruby, which kills any ability for me to modify it. I could learn Ruby, or find a way to bind Python to Ruby, but then would that be the same amount of effort as just replicating Huginn in Python? I don’t know but I know I vastly underestimate implementation times.
Issues
The problem with a lot of existing tools that achieve bits of this ideal, is that free software is not commercially viable. People tend to lock data behind interfaces or APIs. You might be able to extract your data, but you can’t automate it. That’s all okay, because that’s business, but it doesn’t do me any good.
Even without that, getting data is tricky. What do you capture? How? What’s the context? A system where you spend as much time entering data as appreciating it is no good. Natural language processing tools like voice recognition, OCR and entity extraction do an okay job, but maybe not a reliable one. You can possibly get a lot of useful data by passively analyzing externals. For example, tracking home/work times by when you enter or leave the local wifi.
Even with data, what questions do I want answered? Is it easy to ask questions of the data? Is it easy to incorporate new questions and answers into my daily routine? You can record all the weight-loss data you want, but if you don’t have it presented to you regularly, you can’t act on it, so you might as well not record it.
Where to?
I think I need to bite the bullet and learn to build up. Not whole systems. Just stitch little things together over time. Since these are tools for my future, I should have time to evolve them.