Hannah B. Pasandi

Self-Driving LLM Serving

Inference and serving systems that anticipate connectivity, thermal, load, and carbon shifts across device, edge, cloud, and satellite tiers.

Self-Driving LLM Serving

Tether

Personal device mesh routing for LLM inference.
0.2% voice SLO violations22% cost cut10 to 30 second warning

Tether treats the iPhone, MacBook, Watch, and cloud as an inference mesh. It predicts connectivity loss and quota pressure so requests can move before a dead zone or quota wall appears.

move the request before the wall hitsdevicephone·watch·macedgecloudpredictconnectivity loss · quota pressuredead zone ahead
Self-Driving LLM Serving

Drift

Token and network aware routing for LLMs in motion.
300 ms prediction horizon100% availability28 to 37% draft acceptance

Drift keeps mobile LLMs responsive while users move through unstable networks. It predicts connectivity, drafts tokens locally during network round trips, and routes requests across device, edge, and cloud tiers.

draft tokens locally while the network round-tripsdevicedraft tokensedgecloudlocal draftaccept 28-37%prediction horizon ~300 ms
Self-Driving LLM Serving

Spectra

Thermal aware multi tier serving for mobile LLMs.
98.5% SLO attainmentzero generation stalls31% lower energy

Spectra tracks radio state, thermal decay, and round trip time as first class serving inputs. It migrates inference before dead zones and thermal throttling turn into stalls.

migrate before thermal throttling becomes a stallradio statethermal decayround-trip timeservingmigratekeep on devicemigrate to edgeradio · thermal · RTT are first-class serving inputs
Self-Driving LLM Serving

CacheCatalyst

Early cache validation for the latency constrained web.
40% average improvementlower page load timefewer validation round trips

CacheCatalyst moves cache validation into the earliest phase of page loading so unchanged resources can be reused without avoidable round trips. The project treats web performance as a latency problem rather than a bandwidth problem.

validate cache in the first phase, skip round-tripstodayvalidatelateCacheCatalystvalidatereuse unchanged resources, no round-tripmove validation earlier
Self-Driving LLM Serving

ForeSight

Predictive load balancing for mobile edge traffic.
94% prediction accuracy8 microsecond added latency47% lower tail latency

ForeSight forecasts incoming load before traffic reaches the edge, allowing programmable switches to rebalance proactively. It turns flow features into microsecond scale lead time for latency sensitive edge applications.

forecast load before it reaches the edgeflowspredictorflow featuresP4 switchrebalancesrvsrvsrvmicrosecond-scale lead time, proactive not reactive
Self-Driving LLM Serving

Balancify

Adaptive application awareness for datacenter load balancing.
85% lower imbalance1.8 to 3.2x throughput40% lower tail latency

Balancify decides how much application state a load balancer should track. It predicts workload heterogeneity and allocates state only where it improves balance, throughput, and tail latency.

track application state only where it improves balanceload balancerheavy flow → statelight flow → statelessserverserverserverpredict heterogeneity, allocate state adaptively
Self-Driving LLM Serving

SyncCohort

Latency based cohorts for social live streaming.
420 ms intra cohort sync890 ms average latency73% fewer spoilers

SyncCohort predicts viewer latency distributions and clusters friends into synchronized cohorts. It avoids forcing every viewer into a global delay while preserving shared moments inside social groups.

sync friends together, not the whole worldstreamcohort A · 420mscohort Blatency-based