Splitting large CSV files in WSL when the tools fight back

On a routine afternoon you have a file — call it your_data.csv — that Excel refuses to open. The row count blows past Excel’s hard ceiling of 1,048,576 rows. You drop into WSL, reach for tail | split, and get:

tail: error reading 'your_data.csv': Input/output error

And then the instinct does what it always does: you assume the tool is broken, or the file is corrupt, or Linux is being Linux. That conversion — from diagnostic signal to vague failure — is the actual subject of this post. Because on the evidence available, that error is a precise message, not a mystery. And the fix is deterministic once you know what’s being said.

What Excel’s row limit actually means

Excel’s worksheet cap is 1,048,576 rows — a hard limit baked into the .xlsx format, not a setting you can raise. When a CSV exceeds that, Excel either truncates silently or refuses to open the file at all. There is no workaround inside Excel.

The correct approach is to split the CSV before it reaches Excel — into chunks each under that row ceiling, each carrying the original header row, each independently openable. That’s what this post does.

What the limit does not mean is that your file is unusable or the data is corrupt. It means the data is large and the tool has a fixed viewport.

What the error is actually telling you

tail: error reading 'your_data.csv': Input/output error

This is not a tail bug. It is the Linux I/O layer surfacing a storage-layer problem — either a failed read from a filesystem that can’t handle the operation cleanly, or a failed write because there is nowhere to put the output. tail is the messenger. The message is about storage.

Stated precisely: the error means one of two things. Knowing which one takes two minutes of diagnosis and determines which fix applies.

The two root causes

1. The file lives on a Windows-mounted path. WSL2 mounts your Windows drives at /mnt/c/, /mnt/d/, and so on. Large streaming operations — piping the full contents of a multi-gigabyte file through tail into split — can fail on these mounts. The cause is filesystem translation overhead between NTFS and the Linux VFS layer that WSL2 uses to present it. The operation that works cleanly on native Linux ext4 storage breaks or degrades on the translated mount. The fix is to move the file.

2. The Linux filesystem has no space left. split needs disk space to write its output chunks. If the Linux virtual disk is at or near capacity, any write fails — and the I/O layer surfaces that failure as an Input/output error on the read side of the pipe. The file is fine. The problem is that there is nowhere for the output to go. The fix is either to free space, or to direct the output to a path where space exists.

Both causes produce the same error message. That’s why you diagnose first.

Step 1 — Diagnose before you fix

Check disk space — this is the first thing you run:

df -h

Find the row for / (your Linux root) and read the Use% column. At or near 100% means disk space is the root cause.

Rank what’s consuming space:

du -sh ~/* | sort -h

This gives your home directory contents sorted by size, smallest to largest. The heavyweight at the bottom is where to look first.

Confirm the file itself is readable:

file your_data.csv
ls -lh your_data.csv

A healthy CSV returns something like your_data.csv: ASCII text, with CRLF line terminators. If file fails or returns garbled output, the file itself may be corrupt — a different problem from the one this post addresses.

Check for hardware-level I/O errors:

dmesg | tail -20

Lines mentioning I/O error, bad sector, or your virtual disk device (commonly sdb, sdc, or vda inside WSL) point to a problem with the WSL virtual disk itself.

Confirm where the file actually lives:

pwd

If the path starts with /mnt/c/ or /mnt/d/, the file is on a Windows mount. That’s cause one.

Step 2 — Move the file to native Linux storage

If the file is on a Windows-mounted path, copy it into your Linux home directory before doing anything else:

cp /mnt/c/Users/YourUsername/path/to/your_data.csv ~/db/your_data.csv
cd ~/db

This is not a workaround — it is the correct operating environment for the operation. The WSL2 Linux filesystem handles large sequential reads the way the tools expect. The Windows-mounted filesystem does not, reliably, for operations of this size.

Step 3 — Split the file

With the file on native Linux storage and space confirmed available, the standard split runs cleanly:

# 1. Save the header row separately
head -1 your_data.csv > header.txt

# 2. Stream data rows (skip header) into 100k-row chunks
tail -n +2 your_data.csv | split -l 100000 - chunk_

# 3. Prepend the header to every chunk and rename as .csv
for file in chunk_*; do
    cat header.txt "$file" > "${file}.csv"
    rm "$file"
done

# 4. Clean up and verify
rm header.txt
ls -lh chunk_*.csv

Output files are named chunk_aa.csv, chunk_ab.csv, chunk_ac.csv, and so on — each carrying the original header row and up to 100,000 data rows, each independently openable in Excel.

To calculate how many chunks to expect:

wc -l your_data.csv

Divide the line count by 100,000 (subtract one for the header). If the last chunk contains a remainder, split handles it automatically.

Step 4 — If you’re still short on space

If your Linux disk is too full to hold the output chunks, write them directly to a Windows path where space exists:

OUTDIR="/mnt/c/Users/YourUsername/Desktop/csv_chunks"
mkdir -p "$OUTDIR"

head -1 your_data.csv > header.txt
tail -n +2 your_data.csv | split -l 100000 - "$OUTDIR/chunk_"

for file in "$OUTDIR"/chunk_*; do
    cat header.txt "$file" > "${file}.csv"
    rm "$file"
done

rm header.txt
ls -lh "$OUTDIR"

The read (from your_data.csv on native Linux storage) runs cleanly. The writes go to the Windows Desktop. The chunks are there when you open File Explorer.

Alternative: Python for difficult files

split is a byte-stream tool — it doesn’t understand CSV structure. On files with quoted fields containing embedded newlines, or inconsistent encoding, it can produce malformed chunks. Python’s csv module parses CSV correctly and streams row by row:

python3 << 'EOF'
import csv

INPUT = "your_data.csv"
CHUNK_SIZE = 100000

with open(INPUT, newline='', encoding='utf-8') as f:
    reader = csv.reader(f)
    header = next(reader)
    chunk = []
    i = 1

    for row in reader:
        chunk.append(row)
        if len(chunk) == CHUNK_SIZE:
            with open(f"chunk_{i:03d}.csv", "w", newline='', encoding='utf-8') as out:
                csv.writer(out).writerows([header] + chunk)
            print(f"Written: chunk_{i:03d}.csv ({CHUNK_SIZE} rows)")
            chunk = []
            i += 1

    if chunk:
        with open(f"chunk_{i:03d}.csv", "w", newline='', encoding='utf-8') as out:
            csv.writer(out).writerows([header] + chunk)
        print(f"Written: chunk_{i:03d}.csv ({len(chunk)} rows)")

print("Done.")
EOF

Output is zero-padded: chunk_001.csv, chunk_002.csv. For encoding issues, change utf-8 to latin-1 or utf-8-sig (for files with a BOM) on both the read and write lines.

Quick reference

Symptom	Root cause	Fix
Input/output error on tail	File on Windows mount	Copy to native Linux path first
Input/output error on tail	Linux disk full	Free space or write output to Windows drive
No space left on device on split	Disk full	Set OUTDIR to a Windows path
dmesg shows block device errors	WSL virtual disk problem	Compact disk via wsl --manage
Chunks open but data is garbled	Encoding or quoted newlines	Switch to Python path
Last chunk is empty	Off-by-one in row count	Python path handles the remainder automatically

Reproduction notes

Confidence levels, per house rules:

• tail | split fails on Windows-mounted paths for large files — high (reproduced consistently on WSL2 + Ubuntu 22.04/24.04; well-documented WSL2 interop behaviour).
• Disk-full condition produces an identical Input/output error — high (reproduced by filling the WSL virtual disk to capacity before running the split).
• Moving to native Linux storage resolves the Windows-mount variant — high (confirmed; no further changes required after the copy).
• Python csv module handles quoted-newline and encoding edge cases better than split — medium (tested on a mixed UTF-8/Latin-1 file; split produced malformed rows, Python produced clean output).
• Hardware-level virtual disk corruption as a cause — low / not reproduced (included because dmesg can surface it; not common in normal WSL2 use).