An Agentic Harness

Like many others over the last few months I have been thinking about large language models and how they can be used to help our day to day lives by assigning to them tightly scoped recurring tasks. Although I went from someone who has never even heard of LangGraph or how to harness agentic AI only a few months ago I have given myself a crash course on the topic because the area feels fresh with a lot of untapped potential and low hanging fruit.

My project uses everything I have talked about so far in this blog but consigns it to being the back end for a LangGraph ReAct agent. The original idea for this health lake came from a blog post dated many years ago now I had read but wanted to implement myself. Data tooling was still evolving so I believed once I completed my own health lake it used more efficient and a cheaper AWS cloud stack.

The consumer product at the end of this pipeline in the earlier paradigm was a dashboard. At the time of writing dashboards were probably at the peak of popularity and general interest. I believe that in the future this will increasingly be seen as an anachronistic way for users to interact with timestamp data stored in tables. What I believe will replace it is the conversational interface used by large language model providers for users to interact with foundational models in an app or web browser. It is higher bandwidth, and natural feeling that it will just dominate and slowly strangle the notion of consulting and interpreting rapidly updating charts.

However a problem with using large language models as the interface for tabular data is that large language models do not know how to extract table data because data is not text and requires SQL to extract. This to me felt like an unsolved problem however what changed recently is that LangGraph released something called deep agents which gives an opinionated workflow and middleware to an agent to get a SQL query out of natural text to then present the outcome of that query to the user who provided the input message. I feel like this is more momentous than is currently being made out to be because it really allows you to unload cognitive burden to another entity who can work self directed in the background.

The task then was to think about what kind of service maps to this agentic SQL health lake workflow. What I came up with was a health/fitness coach because a coach is someone you interact with and provide fitness data to especially in a platform like TrainingPeaks. In turn a coach returns with suggestions, goals, and motivation.

It was always possible to glue together some API requests and make this into some sort of app and many people have done this. What I think differentiates this is the chat interface. In the future I believe just like in the vision laid out in the Citrini report that chat replaces apps. So chat replaces UI, app layouts, tapping and swiping on glass screens, all of it.

The SQL deep agent is the final piece in terms of asking something out of sample for which the agent has to use intelligence to resolve which is now solved but there are other aspects to a health and fitness coach. These are updating your progress, bringing up any injuries you have so that it can remember and keep that in mind and keeping a store of requests asked all the time for which SQL queries provide answers and it is stored in order to be faster.

Assigning workouts is a key part of a fitness coach agentic workflow but Apple does not allow you to upload any structured workouts to the Apple Watch. You can execute structured workouts you create yourself manually on the watch but that is not an engineering solution. So I was confronted with a walled garden problem and needed to find a solution. My idiosyncratic solution which may well be unique to me is that I wear two watches anyway and the workout can instead be uploaded to the Garmin watch. This also turned out to be a problem because Garmin is another walled garden although it turned out to be a temporary problem with a convenient solution. Although the official API does not allow users to upload structured workouts an online platform called intervals.icu does and the calendar on intervals.icu syncs with the calendar on Garmin Connect which provides a pipeline to get a structured workout onto your Garmin watch to execute from the watch.

Deployment

This is not a commercial product because for a start I am making use of someone elses product (Health Auto Export and intervals.icu) and my system incorporates wearing two watches and there are few people in the world who are big enough dorks to do something like that. However I still want something to show for myself because this has taken up a lot of my energy and time. Therefore I have decided to take my content multi-media. In the age of AI no one reads blogs much less technical blogs because all of the content can be scraped up and served to users who query the subject matters in a large language model chat interface. That is why I have tried to now play the youtube game because content can be served to a wider funnel and if it gets popular I can get more exposure by being algorithmically recommended to others.

It seems to an instinctive response for prospective founders and engineering tinkerers once they have an idea is to pivot to a consumer SaaS and some others in this space have seemed to do so. However my mind from academia (not PhD level but post graduate level) is more naturally drawn to the idea of seeing a problem and finding a solution to that problem so that you can present about it and hope for opportunity that way.

That gets me on to another problem, ideally I would make a github repository so that anyone can replicate my results. However that is a non starter here because we are dealing with personal health data and is there anything more personal and private than that? I can see the rationale behind why Apple by default does not even allow you access to it via a backdoor for fear that once you have it someone else might have it and use it for malicious purposes. Therefore there is no github repository you the audience just needs to follow the video and for some things take my word for it or all the ingredients are straightforward to implement yourself once you get the hang of how each piece works.

In the last few days I have been scouring the internet as best as I can wondering if what I am describing has been done before. After recording my presentation but before publishing this post I began to uncover a hive of activity online that suggested yes indeed many have done so. I would have been somewhat disappointed if my exact setup and motivations and execution were both already done and done better but thankfully I do not believe that is the case. There is some impressive engineering by others not displayed by me but in terms of ideas many are on a similar footing. I will split it into platform specific places of online discourse for both intervals.icu (the forum) and the Health Auto Export app (the discord)

How Crowded Is This Space?

Discord

DrKaizoku

A user named DrKaizoku describes a constantly updating setup that pushes daily gym workouts using an app called Hevy (I am not familiar with this). I had no idea that this was a domain where fitness data helped because admittedly although I am not a lifter my prior notion of progressive overload came from having a diary of your sets and reps rather than sensor data.

I am an occasional strength lifter (twice a week at most at ten minute sessions) but I have never measured it with an app. Instead I use the recorded Apple Fitness Plus workouts. This is because unlike running and cycling where I can follow intervals through my intrinsic motivation I need the extrinsic motivation of someone on screen to get me through the physically challenging part because it is simply not enjoyable.

Intervals.icu forum

IntervalCoach

A user named mrusschen uses whoop sensor data. I have never owned a whoop but this sensor appears to give sleep and things like training readiness. This looks to be firmly in the “app” kind of solution and is a SaaS. The usual app commentary applies here. I think long term it makes more sense to “chat” with the agent to churn the gears of these automations instead of tapping and swiping on a glass screen.

Montis ICU Coach V5 (Railway T2 Engine)

A user named Clive_King designed a specifically ChatGPT large language model first approach which aligns with my vision. He built a ChatGPT app (I have never heard of this concept) where you discuss with the large language model in natural text however it has intervals.icu data only. Users need to make a trade off here between ease of use with a chat interface or annoyance constructing a data driven workflow. My minor gripe with this is whether or not this captures all of the obscure little activities you do throughout your life like pilates, Nintendo Switch peripheral gaming, outdoor soccer etc. That would not be captured all on intervals.icu (unless it has all of your unclassified activity data). Personally I capture all of this on my Apple Watch because as soon as I start doing anything physical my Apple Watch notices something and invariably the software nudges me to record and classify it

Liv

A user named Michael_Skewiderski made the large language model chat interface more of a focus as well as long term planning

ICU Coach

A user named Murat delivered a commercial SaaS implementation that looks to have a lot of features mentioned already and incorporates Apple Health

Conclusion

There is not a lot of entirely new ground I am breaking here apart from the focus on LangGraph at least as far as I can observe. Personally I am glad to see such a thriving community because if there was nothing I might second guess myself thinking this is all a pointless enterprise. All I have right now is LangGraph server running on my Macbook however ideally what would make sense is this workflow gets routed to your personal agent which runs on a home server. Then you talk to the agent just like you would a Clawdbot and you get routed contextually to this “fitness coach”.