Honkin’ File

Had cause today to strain the heck out of vi — a client wanted some known chunk of a 1GB, 5-million-line log file integrated with another log. Took about two minutes to open the file, but once open, was surprised by how snappy it was. Found a buried string in about 15 seconds. Snagged from that point to the end of the file and wrote out to a temp file in about the same amount of time.

:3976053,$ w newlog_temp

Try that in Word.
vi: It’s not just for breakfast anymore!

Music: The Dickies :: Eve Of Destruction

14 Replies to “Honkin’ File”

  1. It all comes down to the right tool for the right job.
    Unfortunately, Word is the wrong tool for any job, but since it’s the only tool many people have, they use it anyway.
    When all you have is a hammer, everything starts to look like a nail.

  2. You could probably do that in UltraEdit, it’s disk based. And more human-friendly than the execrable vi “interface”

  3. UltraEdit is a Windows desktop editor. Lots of desktop programmers text editors could do that. I’m talking here about working remotely on a *nix machine.

  4. mnep — A shorter command would be:

    grep string file.name > newfile.name

    (no need for a pipe there), but either variant merely outputs every instance of string to newfile, which is not the job I needed to do. I needed to take everything in the log after a certain timestamp until EOF, and make that into a new file, i.e I needed the last third of a 1GB file, starting from a precise timestamp.

    There probably is a way to do that with shell commands, but why bother when vi can do it so easily? ;)

  5. Scot,

    I think grahams meant to type,

    tail -n `grep -A string` file.name > newfile.name

    Clever :)

  6. Gilbert, this doesn’t work either on OS X or Linux:

    grep -A string filename

    so the whole command fails. “man grep” does make it look like that would work, but no dice here.

    Also, what would you use for “n” ? Remember, we’re trying to extract from the first occurence of string to the end of the file.

  7. Hmmm… you’re right. I read too quickly on that one.

    Throwing elegance out the door for a moment:

    grep -A `wc -l file.name |awk ‘{print $1}’` “string” file.name > newfile.name

    Yes, it’s an ogre, but it works…

    Going through the whole file to count lines, and then going through it again with grep makes me feel dirty. I’m sure there’s a better way, but I’m too tired to think of it right now.

  8. No problem, it was a neat problem to work with, :) and I don’t get that many opportunities to puzzle over shell problems. I’ve read a lot about *nix, but haven’t really gotten many opportunities to apply the knowledge (other than hobbyist servers, websites, and the occasional mac os x excusion) since the day job is spent administering windows desktops and servers.

Leave a Reply

Your email address will not be published. Required fields are marked *