Macros: To Code or Not to Code

No matter what project I find myself working on, it seems like I have a batch of actions I need to sift through to find all the instances of some conditions and then do something with each instance found at least once a week. You only need to do it once and then move on, so buying a tool or writing some fancy code to handle it seems like overkill. This is where text editors with macro recording functionality can save the day, or at least a few hours of the day.

In this blog, I’ll share a practical use case for macros. I’ll walk you through the example, and then we’ll dive into what it looks like to use macros. I hope that this demonstrates the value and versatility of adding macros to your programming tool belt.

Just a heads up: over the course of the post, I’ll be using Textpad, as its macro usage is easy to use and understand.

Use Case: Error Instances

We can use a macro to look through a large log file to quickly identify and handle error instances.

Let’s say you notice a weird thing that happens on a certain port or slot. Naturally, you want to know everything that happened and when it took place. Here’s where using macros comes in handy!

You can record a macro that finds the first instance of the error and moves it to the bottom of the file. Then, you can run the macro over the rest of the file, and all of the relevant issues will be grouped in one place. Let’s take a look at a tangible example to demonstrate.

A Tangible Example

The content of the file doesn’t matter much, as long as the format and contents are consistent. For some random data for our example, we’ll use a sample Apache HTTP Server log file published as part of Loghub: A Large Collection of System Log Datasets for AI-driven Log Analytics.

We’ll start by making a copy of the Apache_2k.log file and naming it slot13.log to prevent overwriting the original data.

A new file, ready for processing!

In this example, we’ve been told that something happened with slot 13 that we need to investigate further. We’ll start by looking for log instances that may be related. On line 790 we see a notice for Found child 1568 in scoreboard slot 13.

Here is our first instance of scoreboard slot 13.

This seems interesting, but it’d be more interesting to see how many other times this happened and when those occurred. If we search for the full messages, we’ll only find the one instance because the message includes the ID of a child instance (i.e. 1568).

Instead, we’ll narrow our search to scoreboard slot 13. If we repeatedly search for this text we’ll find that it exists in this document three times. At this point, it’d be nice if we could just narrow down our log statements to just those three logs.

Using a text editor with macros, there are a few ways to narrow down to just the data you need. We can move instances to the bottom of the file, we can go one line at a time, or query the outputs to SQL updates. Below, I’ll dive into each of these three methods.

Method 1: Move to Bottom

A simple way to group data is to use a common value that identifies each instance. For example, each of the instances of using slot 13 end with the text scoreboard slot 13, which could be used to find those lines and move them to the bottom of the file with each replay of the macro. For this to work, several steps need to be followed.

Step 1: Pre-Select Identifying Token Text

Find the identifying text that is unique enough to correctly identify the data you want, without being too unique that you’ll only ever have a single match.

In our example above, using Found child 1568 in scoreboard slot 13 was not specific enough because the child identifier of 1568 narrows down the search results to a single instance. On the other hand, searching for a value 13 is too narrow because 13 will be found in a number of other random places, like timestamps, object IDs, etc…

I prefer to use a few words with at least some spaces or punctuation. In this instance, I would select scoreboard slot 13, and then press CTRL + F to set the search value.

Step 2: Start the Recording of the Macro

Start the recording by pressing CTRL + SHIFT + R or Macros > Record. Every keyboard action that you take from now until the recording is stopped will be remembered exactly.

Step 3: Move to the Beginning of the Search Area

When recording a macro it’s important to use as many relative actions as possible. For example, if you are on the 3rd line of the file and you want to go to the top of the file, you might just click the up arrow three times. This works fine now, but later when you’re replaying this macro you may be on line 100, and clicking the up arrow a few times will not get you anywhere near the top of the document.

A better way is to use relative actions. In this case, pressing CTRL + HOME will always take you to the top of the current document. Always moving to the top as the first action helps ensure the data retrieved is generally kept in the same order it was produced.

Step 4: Find the Next Instance of the Identifying Token

Using CTRL + F will take you to the next instance of the pre-selected token from step 1.

Step 5: Select the Relevant Information

Generally, at this point, we’re just narrowing down the instances of the information that we’re looking for, and I just select the entire line of text using Home and then SHIFT + End to select the entire line.

Step 6: Copy or Cut the Data

Now use CTRL + X to cut the data. You might be tempted to copy the data. However, if the data is left without changing it, the next time the macro is run this data will be processed again.

An alternative to removing the data is to alter the text in a way that the search token will no longer match, such as adding an extra space. If the processed file is large, this might be a good point to delete everything before the current row. Since all that data has been checked and skipped, there’s no need to keep it around (as long as this file is a copy of the original data).

Step 7: Move to the Accumulated Data Section

Accumulating all of the responses can be easily done by just pasting the new data at the bottom of the file. Again, you should get to the bottom of the file using a relative keystroke like CTRL + End. If the last line has text, you may need to insert a new line feed to prepare for the next entry.

Step 8: Paste the Copied Data

This one is simple. Paste the copied data at the end of the current list.

(Optional) Step 9: Change the Part of the Line that Matched on the Token Value

As mentioned above, it might be good to change the portion of the text that matches the search token so that this data isn’t processed again later. This can be particularly helpful if you use the To end of file option when saving your macro.

Step 10: Stop Recording and Save

Again, use CTRL + SHIFT + R or Macros > Stop Recording.

Putting It Into Practice

Now that you have recorded the macro, it’s time to use it. You should be able to run it for each instance that you’re trying to find. An additional entry should be added at the bottom. Remember, where the cursor is left when the recording is stopped influences where the cursor will be left each time the macro is played back.

For example, using the steps above, the cursor will stay at the bottom of the file where the last data entry is placed. This has the advantage of monitoring the new log entries as they are added to the bottom of the file.

A helpful variation of this is to record the steps at the end of the recording to move to the top of the document. Then, search for the next instance of the token. This gives you a chance to see what will be processed the next time you run the macro. This is particularly relevant when you want to manually override the macro for specific scenarios.

If step 10 was skipped, you may notice that the number of entries stops growing and the first entry starts moving to the last entry. If you think about it, this is just the macro running. However, now that new entries are not found it finds the first entry in the output list and moves it to the bottom of the list.

Method 2: One Line at a Time

    [Sun Dec 04 17:43:08 2005] [notice] jk2_init() Found child 1568 in scoreboard slot 13
    [Mon Dec 05 04:13:54 2005] [notice] jk2_init() Found child 3755 in scoreboard slot 13
    [Mon Dec 05 10:59:25 2005] [notice] jk2_init() Found child 5568 in scoreboard slot 13

If you don’t like the method we discussed above, here’s another way to do it. The block of code above gives us a group of similar log statements, and we can start placing the pieces together. We’re starting with 3 lines, but this would just as easily work for 50 or 500. This time, we’re recording a macro only intended to modify one line of text.

Say we’re trying to determine which children were found in slot 13 and when. Again, the name of the game is to work in relative and repeatable steps. So to simplify this data one line at a time, I would start with the cursor in the first row of the data, and then I would…

Step 1

Make an absolute move to the beginning of the line (with the Home key) or the end (with End key) to eliminate any variances where the cursor is to start. We’ll start with a line like:

[Sun Dec 04 17:43:08 2005] [notice] jk2_init() Found child 1568 in scoreboard slot 13

Step 2

One extraneous word at a time, remove all the extra cruft until you have just the data that’s needed. Something like:

Dec 04 17:43:08 child 1568 in slot 13

Step 3

Move the cursor to the next line.

Step 4

Stop the macro, and give it a name as needed.

Step 5

Run the macro on the remaining data, and you have a simple output of the data you need. It might look something like this:

Dec 04 17:43:08 child 1568 in slot 13
Dec 05 04:13:54 child 3755 in slot 13
Dec 05 10:59:25 child 5568 in slot 13

Note: It may not be that interesting on 3 arbitrary results from a small file, but it’s much more satisfying when you’re looking through gigabyte logs files to find which users were using the same thread in tight proximity that might prove a new component isn’t thread-safe.

Method 3: Query Output to SQL Updates

For the next method, we’ll take random data and pretend it’s an output of a query that was found to have invalid emails that need to be cleared. We’ll also update an existing field fullName based on the first and last names.

e6253ac4-7b05-4a94-9259-35917f045f42,1,Jon,Doe,invalid,343
f14ff106-3c1b-4c80-a3e6-79b9c0135f16,2,Jack,Doe,invalid.com,341
48b51021-bbce-4922-856c-d8a21c5d8b3f,3,Jason,Doe,[email protected],44
266d63ee-c346-4d4c-83fa-4ce7f14db70a,4,Jane,Doe,invalid invalid,null
d578716e-6bdf-4992-bd34-3c344b82d88f,5,Bob,Doe,invalid@invalid,421
73626710-3c35-44fd-9dfd-f015389db238,6,Dan,Doe,[email protected],null
2110a395-67de-49e5-a590-aeb671157e27,7,Dori,Doe,in.val.id,3
c4df2917-c3e0-4224-8834-eea8662809c1,8,Sam,Doe,in.val.id,443
992fd9df-a920-424f-81f9-28d0d0dc0eef,9,Sara,Doe,invalid at invalid.com,23
1e711475-a42b-4558-bb1b-dd3bafdab829,10,Sean,Doe,invalid,545342

Step 1

Like always, we start with a single row and start recording. Because the initial cursor can be anywhere in the line, we use Home to get to the start of the line.

e6253ac4-7b05-4a94-9259-35917f045f42,1,Jon,Doe,invalid,343

Step 2

We know that the GUID is needed in the WHERE clause, so let’s move it into the next line. Since QUIDs are predictable, we can just select the first 36 characters using keyboard keys only.

Remember mouse movements will not likely be repeatable, but holding down the Shift key and pressing the right arrow 36 times will do the trick every time. An even more efficient approach is to hold the Shift and Ctrl and then use the right arrow to select entire words.

Once we have cut the GUID (and removed the following comma), we can move to the end of the line using the End key and paste the new value at the end. It’s also good to keep common patterns (for example, putting commas between all values), so add a comment before pasting the GUID.

1,Jon,Doe,invalid,343,e6253ac4-7b05-4a94-9259-35917f045f42

Step 3

Now we need to start adding SQL. Let’s start with the general SQL Update statement. We’ll only need the following fields:

• first
• last
• GUID

Start at the Home of the line and remove the first field, so we can add the text to start the statement and use first and last name fields. Don’t forget to relatively replace the comma between the first and last fields with a space.

UPDATE Employees SET fullName = 'Jon Doe',invalid,343,e6253ac4-7b05-4a94-9259-35917f045f42

Step 4

Now we need to figure out how to extract that invalid email text. This time, you noticed that one field had different lengths, number of words, number of punctuation… In other words, this isn’t a scenario where you hold the Shift key and press the right arrow any amount of times and it’ll work for all different types of data.

A better way is to get more created. There are no rules that say working one line of text at a time must always stay on one line. We do have high confidence that we can find the left position of the email field (with all those invalid values). The same can be said about finding the rear extent of the email field. As this so clearly delineates itself into 3 different sections, we can just break up the line into the text before the email, the email, and the text after the email.

UPDATE Employees SET fullName = 'Jon Doe',
invalid,343,
e6253ac4-7b05-4a94-9259-35917f045f42

Step 5

Seems there’s some other field after the email, but we don’t need it so we can remove the whole second line. And after the fullName we’ll clear out the email.

UPDATE Employees SET fullName = 'Jon Doe', email = null
e6253ac4-7b05-4a94-9259-35917f045f42

Step 6

And now this is starting to look like an Update statement. We just need to get rid of the extra comma at the end of the first line and add the WHERE clause on the second line.

As tempting as it would be to use the mouse or just the left and right, it’s much better to get into the practice of starting from the Home or End of the line and holding the Ctrl key down to at least move by whole words. Remember every keystroke that is reported at this point will exactly be replaced later.

UPDATE Employees SET fullName = 'Jon Doe', email = null
WHERE empId = 'e6253ac4-7b05-4a94-9259-35917f045f42'

Step 7

Depending on your plans with the resulting data, you may want to take other considerations into account here. For example, SQL statements where you might grab all of the statements and try to run them in one go, so it might be beneficial to have each statement be on one row and end in a semi-colon.

UPDATE Employees SET fullName = 'Jon Doe', email = null WHERE empId = 'e6253ac4-7b05-4a94-9259-35917f045f42';

Step 8

So running this one macro against the whole list should look like:

UPDATE Employee SET fullName = 'Jon Doe', email = null WHERE id = 'e6253ac4-7b05-4a94-9259-35917f045f42';
UPDATE Employee SET fullName = 'Jack Doe', email = null WHERE id = 'f14ff106-3c1b-4c80-a3e6-79b9c0135f16';
UPDATE Employee SET fullName = 'Jason Doe', email = null WHERE id = '48b51021-bbce-4922-856c-d8a21c5d8b3f';
UPDATE Employee SET fullName = 'Jane Doe', email = null WHERE id = '266d63ee-c346-4d4c-83fa-4ce7f14db70a';
UPDATE Employee SET fullName = 'Bob Doe', email = null WHERE id = 'd578716e-6bdf-4992-bd34-3c344b82d88f';
UPDATE Employee SET fullName = 'Dan Doe', email = null WHERE id = '73626710-3c35-44fd-9dfd-f015389db238';
UPDATE Employee SET fullName = 'Dori Doe', email = null WHERE id = '2110a395-67de-49e5-a590-aeb671157e27';
UPDATE Employee SET fullName = 'Sam Doe', email = null WHERE id = 'c4df2917-c3e0-4224-8834-eea8662809c1';
UPDATE Employee SET fullName = 'Sara Doe', email = null WHERE id = '992fd9df-a920-424f-81f9-28d0d0dc0eef';
UPDATE Employee SET fullName = 'Sean Doe', email = null WHERE id = '1e711475-a42b-4558-bb1b-dd3bafdab829';

In Conclusion

So there you have it! Dealing with a bunch of repetitive actions in different projects can be a real headache, but fear not. In this blog, we took a dive into the practical side of using macros. We walked through an example and showed you how these nifty tools can make your programming life a whole lot easier.

So, why not give macros a shot? They’re like your programming sidekick, saving you time and making your coding adventures way more fun. Happy coding!

Relevant Links

• https://github.com/logpai/loghub/tree/master/Apache
• https://github.com/logpai/loghub/tree/master/Apache#citation
• https://github.com/logpai/loghub/blob/master/Apache/Apache_2k.log
• https://arxiv.org/abs/2008.06448