Seven months ago I ran into a problem: I had guitar riffs that I wanted to transcribe into tabs. I thought I had the solution: take the audio feed, use pitch detection to find the note, then take the video feed to place that note on the guitar neck. That’s the elevator pitch for how TabVision came to be. Entailed below is every step along the way.
Play an E4 on a guitar, then play it again somewhere else. Depending on the instrument there are five or six spots on the neck that produce that exact pitch. A modern transcription model can tell you what you played with better than 90% accuracy. It has no idea where. And “where” is basically the entire point of tablature, which is the notation guitarists actually read.
That gap is what this project was about. TabVision started in January 2026 as what I figured would be a weekend project: point a camera and a microphone at someone playing guitar, get tab out. It turned into 556 commits, four frontends, one formal release, a fourteen-item experiment queue where eleven of the fourteen came back negative, and a decoder whose single biggest improvement came from string stiffness physics I derived off manufacturer spec sheets. Not where I expected to end up.
This is the full account, including the parts that didn’t work, which honestly is most of it.
Act I: The naive build (January)
The first commit is an Electron + Flask demo that drags in a video, fakes a processing job, and shows a tab editor. Within a week it had a real audio pipeline, a video pipeline, and a fusion stage. It also had a set of accuracy targets I would spend months living down: 0.94 single-line accuracy, 0.86 strummed, 0.70 technique detection.
None of those numbers had ever been measured. They were aspirations written in the confident voice of specifications. The project later adopted a rule that a target that has never been measured is not a target, and this is why. When I finally baselined the technique-detection target, it came back at exactly 0.00. There was no technique detector anywhere in the pipeline, and the eval dataset couldn’t even label bends and slides to train one against.
The original plan had three parts. Audio gets you pitch, which is mostly solved, so use an off-the-shelf model. Video gets you position by watching the fretting hand. Fuse them, render tab, done.
The first part held up. The second consumed roughly three months and is one of the most instructive failures I’ve had. The third turned out to be hiding the real problem the whole time.
Act II: The video era (spring)
The reasoning seemed airtight: if audio can’t see the fretting hand, watch it with a camera. I built the whole chain. A YOLO-OBB oriented-bounding-box detector for the fretboard, MediaPipe hand tracking for fingertips, and a geometric map from fingertip coordinates to fret positions, calibrated using the rule of 18 (the geometric series that spaces frets on a neck).
And the lever was real. I ran an oracle probe where I fed the fusion stage perfect string labels just to see what would happen, and Tab F1 jumped to 0.973. If you could extract that information, it was worth nearly everything.
The extraction failed. On real in-the-wild video, the calibrated chain resolved contested strings at 0.574 accuracy. Meanwhile the audio-only playability prior — the thing video was supposed to be rescuing — already scored 0.778 on the same decisions. If you fuse in a signal that’s worse than your baseline, you degrade the result no matter what weight you give it. A learned video model did even worse. A later probe seemed to settle it for good: video was anti-enriched exactly where audio failed, with P(video correct | audio wrong) = 0.285.
Keep that number in mind, because it comes back in Act V, and it turns out to be wrong.
Even setting that aside, two structural problems doomed the approach. The labelled video corpora that exist are 640×360, and at that resolution there is provably no string-level signal to learn; a classifier’s likelihood ratio came out at 1.03, which is indistinguishable from noise. And then there’s self-occlusion. The fretting hand hides its own fingertips from most camera angles at precisely the moments that matter.
So I pivoted. Scope v1 to acoustic guitar, audio-first, and rebuild the whole thing as a disciplined CLI: swappable modules behind strict dataclass contracts, each phase gated on a held-out eval before the next one started. I measured electric guitar rather than pretending it worked (pitch F1 collapses from 0.93 to 0.73, Tab F1 to 0.12 on an acoustic-trained backbone) and explicitly deferred it to a paid fine-tune.
In June, v1.0.0 passed formal acceptance on a sealed held-out player: single-line Tab F1 0.523, strummed 0.676, onset F1 0.94, pitch F1 0.93, faster than realtime on a laptop CPU. Honest numbers, none of them impressive. The interesting part was the error decomposition. Of roughly 380 single-line errors, 322 were the same failure: right pitch, wrong position. The model heard the note and put it on the wrong string.
I wrote in the docs that this ceiling was “information-theoretic,” because the same pitch is acoustically near-identical across strings, so audio fundamentally can’t tell. That sentence sat in the repo for two months. It was wrong, and the specific way it was wrong ended up being the best lesson in the project.
Act III: The experiment queue (July)
After release I ran a structured accuracy program. Candidate improvements went into a queue ranked by expected ROI, one bounded experiment per iteration, and every experiment ended in a written verdict: shipped, or closed-negative with the reason recorded. Fourteen items ran and eleven were negatives. A few of them are worth describing, because the negatives are where most of what I learned actually lives.
Second-opinion model merges. The idea was to merge notes from a second transcription model (MuScriptor) wherever the primary was uncertain. It passed the “is there anything to gain” gate by 3.8×, then failed immediately on arrival: its admitted notes were only 0.18 precise against a derived break-even of 0.528. The derivation was the useful part. The break-even precision for admitting outside notes is computable from your own F1 and match rate, and volume cancels out of the math, meaning how many notes a rule admits never changes the sign of the merge, only its size. Every later candidate got priced against that bar before anything was built. A TabCNN-family retry in August failed the same gates.
A symbolic sequence model. A 414k-parameter transformer trained on 34 million notes of symbolic tab, rescoring the decoder’s candidate lattice. It cleared its control condition, proving that context rather than corpus statistics was the active ingredient, and still missed the pre-declared bar by 0.0033. The miss had structure too: it helped chords about six times more than single lines, and single lines were 77.5% of the loss. So it got banked and closed.
Onset snapping. Align note starts to spectral-flux peaks. This is a published win on piano. Here it actively created timing errors (that bucket went from 15 to 41), because the ensemble’s onsets were already better than the flux peaks I was snapping them to. The published result assumed a detector whose timing was the weak link, and mine wasn’t.
Capo detection from audio. This one got refuted in principle rather than just in practice. A capo at fret 2 produces the identical pitch set to no-capo transposed up two semitones, so no pitch-based detector can ever separate the two cases. The probe recovered 1 case in 60. The capo stays a user-supplied input forever, and I’m fine with that because it’s a proof rather than a shrug.
And then there was Q6.
Act IV: The physics (the one that worked)
Real strings are stiff. Stiffness resists bending, and that pushes a vibrating string’s overtones progressively sharp, so the 10th partial of a note might land at 10.04× the fundamental instead of exactly 10×. The size of that stretch is captured by one number per string, the inharmonicity coefficient B, which depends on the string’s core diameter, its material, and its speaking length. Here’s the part that matters: the same pitch played on two different strings has measurably different B. My candidate pairs differed by 1.6 to 1.8×. Which means the overtone spectrum of a single note carries a fingerprint of which string produced it.
The information was in the audio the entire time. My “information-theoretic ceiling” was actually “features I hadn’t extracted.” A claim about the world had been quietly substituted for a claim about my code, and it survived two months mostly because the stronger phrasing sounded more rigorous.
The build ran as a ladder of gates, and every rung taught something:
Separability precursor, which cost no data at all: every ambiguous candidate pair sits at least 4 frets apart, which is worth a 1.59–1.78× B ratio before any per-string differences even enter. Clearable if B can be estimated to roughly 25%.
The estimator, self-validated on synthetic strings first. That validation caught a real bug: a partial-search window that grew as k^1.5 and was swallowing neighbouring partials by the 10th, returning confidently wrong fundamentals. Invisible on real audio, and it would have been fatal to interpreting any failure downstream.
The classification gates. String classification from B alone hit 0.895 accuracy on hexaphonic pickup data and 0.920 on an ordinary mono microphone, against a count-prior control flat at 0.65. The mic beating the dedicated per-string pickup was the best surprise of the whole project. The pickup is band-limited, and B lives in the high partials. Practically, it meant the channel could run on audio the pipeline already had.
Integration as soft, abstaining evidence. The channel measures its own fit quality (r²) and contributes nothing below threshold, so its failure mode is “no evidence” rather than “wrong evidence.” The error decomposition was one-for-one across 52,000 events. Every gained note came out of the wrong-position bucket, every other bucket moved by exactly zero, and onset and pitch were bit-identical.
Self-calibration failed. Fitting B from the user’s own recording needs about 8 clean notes per string, and a 30-second clip yields around 10 total. Bootstrapping from the decoder’s own labels injects a bias comparable in size to the whole signal. For a while this looked like the ship-blocker, since it seemed to chain the channel to the lab dataset forever.
The portability solve: derive B from published string specifications. B = π³E·d⁴/(256μL⁴f²), where every term is a number you can read off a spec sheet, and the fret law falls out of geometry instead of being assumed. The spec-derived table scored +0.0502, statistically indistinguishable from the dataset-fitted table’s +0.0525. At that point the training dataset got demoted from source to test set. Nothing was fitted to anything.
The full-development gate came in at +0.0443 [+0.0339, +0.0555] over 300 clips with the config frozen before the run. Sealed-player confirmation: +0.0780, and +0.1396 on the solo tier — a 25% relative gain on exactly the tier I had declared information-limited two months earlier. It shipped as the default for clean steel-string acoustic, with a domain guard proven by unit test rather than measured by eval run: nylon, capo, and alternate tunings abstain by construction, since a wrong table would be worse than none.
A robustness study afterwards mapped the failure envelope across 17 pre-declared table perturbations. String gauge and scale length turn out to be noise. The entire risk concentrates in wound-core construction, which is the one spec manufacturers don’t publish, and the gain lives almost entirely on the four wound strings.
The capo work rode the same wave. Routing declared-capo sessions through a coordinate-shifted (“covariant”) prior recovered what turned out to be an outright collapse, 0.296 → 0.683 at capo 2. The old behavior hadn’t been a shortfall so much as the decoder breaking outright when every playable candidate sat above the capo.
Act V: Humility, measured (late July)
Three episodes from the same two weeks, all of them corrections the process caught before I could publish something wrong.
The headline was the luckiest player. The +0.0780 sealed result went into the README as ~0.73 aggregate. When I then measured all six players, the channel’s gain ranged from +0.047 to +0.101, a factor of two, and my sealed player happened to sit at the maximum. Nothing was mis-measured. A single held-out set protects you against overfitting, but one draw from a varying population can’t tell you an effect size. The headline got re-based downward to 0.66, the sealed player got rotated, and the README now explains why.
The sealed set also earned its keep directly. A uniform level correction to the physics table, supported by three independent measurements of the same error, scored +0.0160 on development data and −0.0066 on the held-out player, with non-overlapping intervals. It had already been built, tested, and written up as a shipping candidate. One sealed run reverted it. The effect was physically real but instrument-specific, and development data structurally could not see the difference.
And the video verdict got corrected, in both directions at once. Remember the 0.285 “video is anti-informative” number? Re-running the identical probe after fixing a fret-mapping bug (a unit-neck coordinate was being projected to fret 24 instead of through the calibrated map) moved it to 0.763. Video was never anti-informative. It was mis-projected. And yet the end-to-end verdict still stood, because against real audio predictions the corrected video channel moved Tab F1 by +0.000836 with a confidence interval touching zero, and a probe showed even gold position windows could only add +0.00009. The decoder’s surviving candidates mostly produce the same tab either way; there was almost nothing left for a coarse position window to fix. So video ships as opt-in, on evidence of negligible effect rather than evidence of harm. That distinction matters, because the buggy framing would have permanently closed a lever that is merely weak.
Both of the project’s worst errors — “the ceiling is information-theoretic” and “video is anti-informative” — were confident, well-written, and supported by real measurements. What caught them both was the same habit: insisting on being able to state the mechanism, not just the number. A result whose mechanism you can’t articulate is a result you haven’t finished checking.
Act VI: The pivot that stuck (August)
A five-track parallel program priced out everything that remained, and the most interesting output wasn’t a shipped feature. It was a price list.
A timbral string classifier had a large ceiling (+0.19 oracle) and a real signal (AUC 0.71), and the conversion was still impossible, because weak per-note evidence washes out against a prior that’s already at 0.65 on the same notes. Closed for the third and final time, this time with the mechanism written down.
The detection buckets were a mixed bag. The attractive “harmonic leakage” story about spurious detections turned out to be a base-rate artifact. Fifths and fourths are 29.6% of false detections, but they’re 37.2% of all intervals in this music, so the enrichment is 0.8×, which is to say anti-enriched. Missed notes, on the other hand, showed real structure: missed at 1.6× base rate inside dense chords and on very short notes, at half base rate when isolated. One bucket closed, one legitimate build candidate opened.
A personal position prior came in at +0.0305 [+0.0183, +0.0430]. Give a player a prior built from their own playing and accuracy rises for all five players tested, most for the most idiosyncratic one (+0.076). Players genuinely differ in where they play the same notes. That’s a small empirical fact about guitarists, and it ended up becoming the product direction.
Because this is where the failed video chain finally found a job. FretCam, the live fretboard HUD built out of the video era’s wreckage, has a measured profile of “position on 27% of frames, correct on 100% of them.” That profile is wrong for a decode-time witness, which needs coverage, and exactly right for a labeller, which needs precision. So the camera’s role inverted. Instead of arguing with the decoder in real time, it silently harvests confirmed (pitch, string, fret) labels whenever its evidence pins a unique candidate.
The studio app closed the loop end to end. Transcribe a take, fix it in the review UI (every note is confidence-graded, and the doubtful ones get queued for one-keystroke triage), press Bank gold, and the corrected take becomes perfect training labels for your personal prior, since a human-confirmed correction needs no alignment guesswork. The system that spent seven months learning what it couldn’t detect now learns you.
The scoreboard
Datasets: GuitarSet (the workhorse — 360 acoustic clips, mono mic plus a hexaphonic partition that validated the physics estimator), GAPS (classical, the cross-domain gate), EGSet12 (reproduction checks), Guitar-TECHS (the electric measurement that scoped v1), SynthTab and PDMX (symbolic corpora, both closed negative for priors), and the 640×360 video corpora, which are structurally insufficient for string resolution.
Models that shipped: the highres-ensemble audio backend, a Viterbi playability decoder, GuitarSet- and GAPS-trained position/sequence priors, the spec-derived physics table, a capo-covariant prior transform, and the personal prior builder.
Models that didn’t: the YOLO-OBB + MediaPipe video chain (opt-in only), a learned string-resolver CNN, a fret-keypoint model that lost to its own geometric baseline by 0.089, a 414k-parameter symbolic transformer, two TabCNN variants, MuScriptor as a second opinion, an n-gram corpus swap, and every form of per-instrument calibration.
Roughly one success per ten attempts, each failure cheap, each verdict written down. That ratio is the method. The single success was worth more than everything else attempted combined.
What I’d tell you if you’re building something like this
Decompose errors before hypothesizing. “Accuracy is 0.52” suggests nothing. “322 of 380 errors are the same pitch on the wrong string” is a research program.
Price the oracle first. Before building any estimator, compute what a perfect one would be worth. A perfect per-player calibration was worth +0.0027, which killed a multi-week build in one afternoon. A perfect video window was worth +0.00009, which ended the video era with arithmetic.
Don’t let a claim about your features masquerade as a claim about the world. “Information-theoretic ceiling” cost this project two months and nearly 0.10 of accuracy.
Sealed data is a tool you spend, not a formality. It reverted a shipping candidate that three independent measurements supported, and it was right to.
Write down the mechanism of every result, positive or negative. Both of my worst errors were caught the same way: a number I couldn’t attach a mechanism to turned out to be a number I hadn’t finished checking.
Failed components can succeed in a different role. The video chain was a bad witness and is a good labeller. Precision and coverage are different products.
The pipeline today transcribes clean acoustic guitar at ~0.93 onset F1 and ~0.66–0.68 tab accuracy on players it has never seen, faster than realtime on a laptop CPU, and it knows, with a written measurement behind each item, exactly what it cannot do. In applied ML I’ve come to think that second property is the rarer one.
The repo’s docs/EVAL_REPORTS/ directory contains the reproducible run behind every number above; docs/DECISIONS.md is the full decision log, negatives included.