Double Down on Scaling Channels

Don’t diversify channels. Double down on growing channels.

There’s a school of acquisition thought that it’s better to cap investment in a specific marketing channel to reduce the risk to the business. This is particularly true when running acquisition on algorithmically driven platforms, e.g., Facebook.

Instead double down on what’s working. If a channel has doubled in acquisition volume, it can be done again. Double a channel 7 or 8 times, and now there’s a very solid path to scale.

I’ve scaled acquisition from 5 customers a month to over 2k but continuing to repeat what was working.

Businesses are valued on revenue levels and revenue growth.

Double Down Within a Channel

It’s possible to down down within a channel, not just on a channel.

This plays out on a high level (which channel) as well as a micro level (which targeting or audience). Once a channel has doubled, it can almost always double again. I’ve had the same keyword go from a handful of conversions to 100 conversions a month.

The levers for increasing gains are almost always the same:

  • More impressions
  • More clicks
  • More conversions
  • Higher lifetime value

This often translates down into:

  • Targeting (keywords, audiences, national v local)
  • Creative
  • Landing experience
  • Pricing
  • Merchandising (whether it’s a good or service)
  • Retention efforts

Running the growth process to double again is even easier than doubling the first time because the team has built out more processes. Come up with the growth plan, the hypothesis, and the testing structure.

Diversify Opportunistically

Develop new channels. Test out new paths to scale. Do it in a systematic way driven by opportunity not by fear. An opportunity driven approach is formed with a solid hypothesis on why a certain channel should work:

  • New product experience that enables channel market fit
  • Competitive insight shows that a similar company that’s successful
  • New platform features that serve your business well
  • New LTV that opens up previously expensive channel
  • New attribution methodology that makes it possible to judge success (my favorite)

Diversify, but do it with a specific reason on why it will work and then try it out.

Dealing With Algorithm Changes

Algorithms change. Acquisition numbers will fluctuate as algorithm changes, sometimes by up to 50%. This is a scary feeling. Watching a dashboard show a 50% drop and then explaining that to leaders in the company. It creates an internal sense of panic.

Act with a clear, calm plan in this case:

  • Gather the data – understand what was impacted, by how much
  • Gather related writing – many platforms share insights on what drives decision making
  • Reach out to leaders at other companies – were they affected? how much? what theories do they have
  • Try reaching out to relevant leaders AT the platform company (Google, Facebook, Apple)

Come up with the plan on what to fix. Explain why it’s important, the reasoning behind it, and go out and execute. Sometimes big, scary changes might be needed. It’s worth it – they can always be undone.

Service Business >Tech Enabled Services

Services Have A Larger Market

Service businesses are better businesses than Tech Enabled Service (TES) businesses. Service businesses can charge higher prices points for a similar service as a Tech Enabled Service businesses, and serve a wider set of customers. Effectively creating a larger market.

Services Have Higher Operating Margin

Service businesses can have higher gross margins. Service businesses can drive down operating costs through better processes, efficient use of labor, and decreasing software costs. Software costs decrease over time through the commoditization of SaaS products.

At the same time, Tech Enabled Services are saddled with the high salary cost of R&D without the corresponding benefits. The software platform developed for a TES does not achieve the same leverage as a normal software company – because there is one target customer of the platform. Gross margins are low. And, a TES doesn’t benefit from the commoditization of SaaS that drives down software costs over time. Instead, R&D costs are more like operating expenses.

Ultimately, this reduces the profitability of the TES and limits the available moats for the company to defend itself against competitors. A TES can be picked off by well run service companies or a pure software company.

Services Can Charge More

Service businesses can charge higher prices than Tech Enabled Services for a variety of reasons.

  1. Services anchor their price point at hiring a full time employee – leading to a higher price.
  2. Services can handle a wider array of situations – if something doesn’t fit inside the box, the company can just chart more for it.
  3. Services can used the past resume/eduction of the team to price the service – think McKinsey

High Operating Costs

Service businesses are labor costs plus pricing. SaaS companies are server costs plus customer success. Technology Enabled Service companies are labor, server, customer success, AND the costs of developers to build the technology.

Neither Services Nor TES Have Moats

A number of traditional moats are unavailable to TES.

  • Low cost producer – the higher operating margin of R&D means the company cannot be a low cost producer. At the same time, because a TES can only scale as well as the services component, the TES cannot achieve a massive scale.
  • Network effects – The service is provided by the TES. This reduces the network effects that might come with multiple parties using the same software, i.e., Google Docs
  • Data – This seems like one of the most likely moats, after all, the TES is providing the service, capturing all of the information, and can harness that data. A16Z addresses the Empty Promises of Data Moats in this great blog post.
  • Brand – Brand is expensive to build without a great product experience. And, with a service, the user experience is going to be highly variable and dependent upon the service provider.

Build Software or Build Services

It’s tempting to think that a service business can be improved and scaled like a software company through the use of software. This is an illusion. Software can improve the productivity of the services provider, but it’s better to use off the shelf software. This will keep operating costs lower over time. Or, just build software for the service providers in the industry.

Related TES Articles

Staffing Firms Recovering Market Cap


A previous post showed how long it takes staffing firms to recover revenue after recessions. Noting the unexpected similarity in recovery time between the dotcom crash and the Global Financial Crisis (GFC). importantly, that ⅓ of staffing firms never get back to the same level of revenue.

There was a noticeable difference in how long it took staffing firms to recover market cap between the dotcom crash and the global financial crisis. Despite the severity of the Global Financial Crisis, firms recovered market cap more quickly after the global financial crisis than the dotcom crash.

After the dotcom crash, it took 9 years on average for staffing firms to recover their market cap, as compared with 4 years after the Global Financial Crisis. Albeit, post dotcom crash, 30% of firms never reached their market cap highs versus 20% after GFC.

Two notable differences between dotcom and the global financial crisis (GFC) are the extent of the unemployment and the aggressiveness of the Federal Reserve’s responses.

In the dotcom crash, unemployment went from ~4% to 6%. Employment took 5 years to recover to pre-crash levels.

The global financial crisis went from 4.4% to ~10% unemployment and took 7 years to get back to pre-Financial Crisis employment levels. 

The Federal Reserve responded to the GFC aggressively: buying up assets and lowering interest rates dramatically.

Federal Reserve Balance Sheet Over Time

Federal Funds Rate Since 1998

The bullish case for staffing firms coming out of this pandemic is that there is, unfortunately, ample opportunity around hiring. With 30M+ Americans out of work, the hiring opportunity has never been greater.  This, paired with significant activity from the Federal reserve could spell a quick recovery for Staffing firm stocks.

The bear case has two main factors: workers get directly re-hired by old employers or workers never re-enter the workforce. 

Companies that see an uptick in growth and directly re-hire their previous employers do not need an employment agency, although there might be opportunity in coordinating the mass re-hiring. There will also be a large set of the currently unemployed that never return to the labor force. 

Those that never return to the labor force will not return for a number of reasons, and those reasons will vary over time. 

In the short-term, unemployment benefits and eviction moratoriums are more compelling than risking a Covid-19 infection in the workplace. Some workers will opt for freelance/self-employment and turn to alternate platforms. 

To date, though, publicly traded Staffing Firms are lagging the S&P 500. Most staffing firms are down about 25% from pre-pandemic levels, while the S&P 500 is down only about 10%.

Staffing Growth Marketing Teams to Products

There is no right structure. There is a structure for right now.

The growth marketing structure for a single product company is straightforward. As the product offering grows in complexity so does supporting the products.

A good team structure can lay the foundation for growth marketing team and product success. The wrong one will create employee churn, channel conflict, and lower Return-on-Ad-Spend (ROAS). The right structure depends on the maturity of the products, the strategic marketing capability of the product team, and the overlap in target customer base.

The basic structure options include: a central queue, dedicated team member, embedded, and an agency model.

Central Queue Through Growth to Multi Product Company

Shared services, or a central marketing queue, is great with a central capable team and stakeholders that are less experienced in growth marketing. This happens as the company transitions from a single product (or market) organization to a multi-product or international rollout.

The marketing team supported the dominant product and the company needs to experiment with new products to grow revenue. The fiscally responsible solution is to carve off time from the existing marketing team to support the new product. If the product goes well, the team can hire a new resource. If the product doesn’t see adoption, there’s no need to let people go.

As things grow (either the introduction of multiple products or the new product does well), the team has to decide how to prioritize between products. This is the key challenge of this model.

If the product is very successful, the option of embedding a marketer will be discussed.

Embedding Creates Attrition

With a central team and a successful product, there will be tension over speed of execution and knowledge of the product. Embedding a marketing employee from the central queue into the business unit becomes an option. The business unit will want to move faster and work with individuals deeply knowledgeable about the product. If done this creates a matrixed structure.

Matrix structures create conflict. Employees are strung between the strategy of the business unit and the execution of the central team. In order for this to work, the marketing lead needs to defer on strategy and just focus on managing the team members execution.

This can be stressful for all parties and requires constant communication between all parties involved. At some point, just moving the marketer entirely into the business unit organization comes up. This creates a different set of challeneges.

GM Models Mean Dealing With Crossovers and Channel Conflicts

Successful products eventually graduate into their own business structure under a General Manager (GM). The GM role splits functions out of a central area and into a new business unit. The entire unit moves cohesively and allows for more cross functional thought on the business unit. The downside with marketing, like other areas, this creates conflict around shared channels, messaging, and brand,.

Specifically, with digital channels, campaigns from products will compete against themselves. Product ‘A’ and Product ‘B’ will bid against each other. Product A might have superior unit economics but product B might have authorization to run at a more aggressive Return on Ad Spend (ROAS) in the growth stage of the product. Managing this conflict is a key challenge of this model. Especially if there is no longer a central marketing function to make determinations.

A lack of coordination also creates missed opportunity. Advertising one product will create a halo and bring in users for a secondary product. WIthout the right systems, this additional value will be missed and the company will under calculate ROAS.

Internal Agency – When Execution Is All Thats Needed

In the agency model, the business unit tells the marketing team how to act. This structure is all about execution and requires a strategic marketing partner that is part of the business unit. This structure works best with an experience strategic lead in the business unit that has done marketing or worked closely with marketing in the past. And, the “agency” part of this

If the strategy marketing partner is missing, this is a bad structure. If the “agency” side of this includes accomplished marketers that want to execute their own vision. This is a bad structure.

With either a business unit lead that has limited marketing experience or a skilled marketing team, instead consider a shared services model.

This model is viable with a large central team and multiple products. It solves some of the issues of channel conflict and matrix structures, but leaves product knowledge and execution speed as open items to resolve.

Adapting Over Time

Structures change and evolve both as the product offering evolves and as personnel evolves. Organizational structures often grow organically and diverge from an idealized structure. As this happens, the ideal way to staff a growth marketing team to a product will change as well. That’s okay. That’s normal. It’s better to adapt to the new environment than trying to maintain an old structure.

Incrementality Testing: True Value of Your Ads

Why Test for Incrementality

Understanding the incrementality of advertising spend is one of the biggest advertising efficiency step changes there is. Bigger than creative. Bigger than multi-touch attribution. Despite that, incrementality tests are rare: rare to discuss and rare to implement. All of the biggest and best advertisers run them.

Incrementality tests are not limited to display campaigns and Public Service Announcement (PSA) tests. They can be run with any creative type (brand v. direct response) or medium (including search advertising).

Incrementality tests help with more than just the incremental cost per conversion. They also help understand if advertising helps with re-activation and retention.

Some of the largest retention gains come not from email campaigns but from mass media campaigns. Continuing the emotional connection a user has with a product is an underappreciated way to retain a user. 

Incrementality Tests and Cookies

Testing bifurcates around cookies. Tests can use cookies or be agnostic to cookies. Tests that use cookies will have greater power, can run with smaller sample sizes, and work with a smaller lift. 

Cookie methods are the easiest to implement, but are limited due to user behavior (switching devices, deleting cookies) and overall changes in user privacy settings and regulation.

Tests without cookies work regardless of changing privacy controls, cookie deletion, multi-device use, and can pick up effects on other parts of product usage such as re-activation and retention. Re-activated and retained cookies might not be part of an acquisition incrementality test, and so will be missed. 

Overview of Test Types

Test TypeBenefitsConcernsWhen To Use
Geo TestWorks regardless of changing privacy settings around cookies or multiple devicesStatistical powerNon-digital tests or when digital spend is sufficient to detect at scale
A/B TestWorks on smaller ad budgetsIssues around cookiesDigital tests with low lift
Public Service AnnouncementAd platforms make these easy to doTrue incrementalityDisplay campaigns when other options aren’t available
MultivariateSee the holistic benefits of multiple advertising channelsTest setup is complex, more expensive to runTo understand the benefits of multi-channel campaigns.


Geo Tests 

Geo tests get around privacy issues related to cookies, and multiple devices, and work across a broad set of advertising medium: TV, out-of-home, radio, SEM, and display. Setting up the tests requires understand what targeting options are available for the media, and if that inventory is comparable (For example, National v Location TV options). 

It’s important to validate that the geo groups had good correlation for some period of time before the test intervention is run. 

Once groups are selected, the test is run for the appropriate period of time. Factors that influence test length are:  time to convert, time for media to ramp up, minimum contract lengths, etc.

Choosing a greater number of smaller geos can provide a more robust test than a smaller number of larger, matched markets. This is because a test of say, NY vs LA could be thrown off by differences in situation in NY (maybe a hurricane in NY, for example). More pleneitufl, smaller geos, helps insulates against the impact to one geo.

This type of test can be analyzed in a number of ways. A traditional hypothesis testing framework could be used with sufficient information. For example, if the test is done through a digital ad network that provides impressions, clicks, and conversion data. 

Another method is to use causal inference, and a tool like Causal Impact, to compare test and control geos. One of causal inference’s strengths is that it can get around traditional limitations of cookie based testing (multiple devices, privacy controls, cookie deletion). 

A/B Tests

A/B tests can apply to several advertising activities: retargeting, email, and seo. A/B testing for retargeting and email purposes is identical to a typical website a/b testing. A control group will receive no treatment (email, display ad) and a test group will receive the treatment. The test results can be analyzed using typical hypothesis thing techniques. 


Public Service Announcements

Display tests are pretty common, although this feels like a more old school method. Every user gets cookies for the test, although a portion of user s will see a public service announcement instead of the advertisement. 

Multivariate Advertising Tests

A fancy way of saying multiple types of advertising activity on the same users. This could involve a control (no treatment), a test with one advertising type (display), and another method online video (OLV). The goal here is to determine whether multiple advertising media and messages have additive benefits that produce more efficient results than just one advertising method alone.

Multivariate tests, generally, are more complex to do well than a/b tests. Setting these up in an advertising environment takes more care and coordination as well.

Getting It Right

Incrementality testing illuminates the value of advertising, which helps with medium planning and creative. It also helps with budget allocation. Testing can be done with any number of advertising channels and is relatively straightforward.

Are you doing incrementality testing today? If not, which method do you think you could get started with first?


Click to access kellogg_fb_whitepaper.pdf

Click to access 45950.pdf

Staffing Firms Revenue After a Recession


In an analysis of 12 publicly traded staffing firms (job advertising, placement, and temporary workers) through the Dotcom bubble and the Global Financial Crisis, the behavior of staffing firm’s revenue was remarkably consistent.

  • Roughly ⅓ of the publicly traded firms never regained the same level of revenue as before the start of either recession
  • The average firm saw year-over-year declines in revenue for 7 quarters (1.75 years)
  • Of the firms that regained revenue, the median time to retouch revenue highs was 13 quarters (3.25 years). The average was slightly longer. 


The data encompasses 12 companies, with earliest recorded revenue starting in 1996 and goes through Q4 2019. The following stocks were analyzed:

  • MWW
  • RHI
  • MAN
  • ASGN
  • AMN
  • DHX
  • KRFC
  • JOB
  • MHH
  • RCMT
  • TSRI

30% of Firms Never Recover Revenue Highs

Out of 9 publicly traded staffing firms in 2000 and 12 during the Financial crisis, ⅓ never regained their previous revenue highs. Some, like Monster, came close after the dotcom bubble, only to have revenue decline when the Financial Crisis hit. 

SummaryDotcom CrashFinancial Crisis
Re-reach Revenue High68
Never Regain Revenue Highs34
% Never Regain Revenue Highs33%33%

An example includes Monster Worldwide. Monster’s revenue peaked at close to $1.5B in 2000, and made it back over $1B before the 2008 Global Financial Crisis. Revenue mostly declined until 2016 when it was purchased by Randstad for $430. 

It takes about 3 years for Revenue to Recover

For firms that regained revenue highs, the median time to regain revenue was about 3 years. The average was slightly longer than the median for both recessions. GEE Group (ticket:JOB) took 54 quarters to return to pre-dotcom revenue levels, creating significant skew.

Time to RecoverDotcom Ave RecoveryGFC
Median in Quarters1313
Mean in Quarters1716
STDEV in Quarters156
Median in Years33

Revenue shrinks for about a year

After both recessions, the average firm saw revenue decline year-on-year for 7 quarters, or about 1.75 years. The industry, overall, took 6 quarters to start growing revenue again. 

Recovery After the Financial Crisis Has Been Tepid

Since the end of the Financial Crisis, revenue growth for the listed staffing firms, overall, has been weak. Four firms IPO’d between the Dotcom crash and the Global Financial Crisis, this pulled up the post Dotcom crash year-on-year revenue performance numbers. Monster was acquired in 2016, and this exit will bring down the 2017 year-over-year numbers, slightly.


LinkedIn filed to go public in 2011, but reported revenues from 2007 on for their S-1 filing. The Financial Crisis likely had an impact on their revenue growth. Revenue from 2007 to 2008 grew 100%. Revenue from 2009 to 2010 grew 100%, but revenue from 2008 to 2009 grew only 50%. 


The Covid-19 Pandemic created an unprecedented level of unemployment in a historically short time period. This recession is not like the previous two recessions. It’s difficult to project how staffing firms will perform, but it’s reasonable to think a certain percentage (maybe a third?) will never regain the same revenue levels. 

Fast growing, profitable, low debt firms are in the position to emerge the strongest, but even that, regaining lost revenue might take at least a year, if not more. 

Big Companies > Startups

Preview(opens in a new tab)

It was the 28th consecutive day of sun. My boss had just made two female coworkers cry over how little he paid them. Little Woodrow’s was empty because it was 3 PM on a Wednesday. We were drinking. It was part of the culture. 

“We’re going to IPO!” my boss said. As a 22nd year old reading stories of startup success in Wired, and not really grokking how the dotcom crash of 2000 would affect IPOs in 2005, or business fundamentals, I was all in. Convinced that a company doing $2M a year, growing 30% a year, losing money was going to IPO. 

I had lucked into the promised land of startups. 

A generation, or three, of Americans has entered the workforce dreaming of startup riches. Steve Jobs, Jeff Bezos, Mark Zuckerberg, Evan Spiegel, and others had created visions of wealth that were the wildcatter oil fortunes of yesteryear.

The early employees of the tech companies, though less touted, also were able to obtain vast sums of money, creating independence, happiness, and additional financial opportunities. Founders and VCs were happy to oblige the picture, talking about the unlimited financial potential of startup equity. 

It’s not all rosy. Most evangelists will acknowledge that startups are hard (depressions, founder burnout) and that 90%+ of startups fail to have a meaningful exit (not to mention the importance of understanding liquidity preferences), but that’s often pitted against other startup benefits. The ability to make a difference. That working in a corporate is soul crushing drudgery. Learning on the jobs versus learning from corporate processes.

The 9-5 corporate job boosters are few (or non-existent).

I’ve co-founded 3 companies. 2 of which eventually had an exit. I’ve worked for 4 other startups. 4 of which have exited (none have failed). And, I’ve been at one fast growing tech company for over 6 years now. 

Founders and VCs are Lying to You

I joined Indeed in 2010 when it was around 200 employees. Indeed, nearly 10 years later, there are approximately 10k employees.

I left Indeed from 2013 to 2016 to start Experiment Engine, so I can’t claim to have seen the entire transition, but it’s been pretty amazing. And it creates amazing opportunities for employees that stay.

There are numerous personal and professional benefits of being part of a larger, growing company. 


I’ve worked with a number of individuals for years. I’ve seen them grow and mature as adults. Get married, have kids, and enjoy life. On the other hand, because Indeed is growing, I’ve been able to work with individuals from some of the best big companies and less successful small companies. There are thousands of interesting people that share being an Indeedian. 

Employee Development

Larger companies establish teams of individuals focused on developing the employee base. Internal programs include training on managing, leadership, communication, data analysis, seo, marketing, and more. 

Companies also provide thousands of dollars a year (maybe $10k?) for tuition reimbursement, conferences, and other learning materials. 


Big companies might move slow, but the average employee will have multiple projects under their purview. At Indeed, this could include launching a new product, working to improve the go-to-market of an existing product in an emerging market, and continuing to position a late stage product for market leadership. 

Risk Taking

It’s easier to take personal or professional risks in a large company. Large companies typically have higher cash compensation (salary + RSUs) versus a pre-IPO company. This makes it easier to take personal risks (angel investing or working on a side hustle). 

Large companies typically have greater ability to absorb failure on employee projects, as well. This means most projects are trying to achieve the best outcome possible. 

International Travel

In order to grow, Indeed became an international company with tech offices in 4 countries and sales offices in a large number more. Connecting with co-workers might require international travel to locations like Tokyo, London, Sydney, New York, etc.

VCs and Founders will sing the siren’s call of money, making a difference, and doing something new. Just know that big companies offer all of those benefits plus everything I mentioned above.

Preventing Wasted Crawls Part 1 of Many

Googlebot loves to crawl: it’ll crawl any thing that looks like a URL, anything it can find in javascript, html, or on the page. If it looks like a URL, Googlebot will try to crawl it. Great for Google, probably great for web users because Google learns more about the web, but it can lead to wasted crawls for web owners.

As I mentioned in a postion about initial SEO decisions for The Dog Way, I blocked all category pages. I’ve loosely monitored Google’s crawls and found them just crawling any available URL: pagination, sorting by size, price, color, et al.


(For those that look at log files, you’ll notice the IPs aren’t Googlebot IPs, we’re using Cloudflare to try to speed up the site and all requests come through their IPs.)

Now I need to find all of the URLs I should have blocked, but didn’t. Very, very simple unix command: wget -O- url | grep urlpath.*\” | sort | uniq

(What I actually ran: wget -O- | grep -o dog-boots-and-shoes.*\” | sort | uniq)

Here’s what the output looks like:


To break that down.

wget -O- url: wget is a program to download files, by default it’ll save the file in the current directory. The -O- tells it to redirect to the stream output. The url is the url to download.

grep -0 urlpath.*” : urlpath in this case is the the part of the URL after the domain. In this case it was dog-boots-and-shoes.*\” (the \” is a way to escape out the ” to treat it as only one “). The ‘-o’ outputs just the text that matches and nothing else.

sort | uniq : sorts the lines and then just outputs the unique ones.

What were the key takeaways from that? I need to improve the handling of pagination, prevent Google from crawling: limit, dir=, size=, and color=. The /p/ are products and are already blocked in robots.txt. The really simple changes are to just block them all in robots. Other options are configuring URL parameters in Google Webmaster, trying to block the links out through rel=”nofollow”, or, for pagination, using rel=”next” and rel=”prev”. For right now, I’m just going with robots.txt because it’s the fastest way to fix the crawls.

Initial SEO Decisions for The Dog Way

Quick Discussion of SEO For The Dog Way

If you spend a few minutes looking around The Dog Way, you’ll notice there is almost no attempt to optimize the site for search engines. Further, almost all of the content is blocked in robots.txt, and in fact, until three week ago, the entire site was blocked. Two reasons.

The first is that since the products come from drop shippers found through there is no original content on the site.

The second is that the main focus of customer acquisition will be through social media channels, hopefully.

Having said that, there will still be some attempts at SEO, and the plan looks like this.

1. Allow bots to crawl the home page, about us, and blog.

2. Do some keyword research to decide which keywords to target through category and sub-category pages.

3. Write entertaining, good content on the category pages, and then the sub-category pages. As each page gets content, I will unblock it in robots.txt and then submit it to Google to crawl.

4. Focus on image and video optimization after that. Dogs are cute, pictures of dogs are cute, people like clicking on them, so I’m hopeful about the last tactic.

One question that comes up with blocking an entire site in robots.txt is how long does it take to get re-included and re-crawled. Turns out that Google still checks the robots.txt everyday, even when it’s blocked. You can see this by going into the logs and looking at the crawls.

Hosting for The Dog Way is done through SimpleHelix and the keep at least one day of log files on the shared, apache server under the symbolic link access-logs. Here’s how I monitored the crawl activity.

Step 1: cd access-logs

Step 2: nice grep “Googlebot” | more

This allows me to pull out all page requests from Googlebot and then page through them a few a time. SimpleHelix updates the past 24 hours and always stores it in ‘’, so there was no need to specify a date or anything else, this isn’t always the case. Grep is a unix utility for looking through files.

I unblocked Google from the homepage about three weeks ago and they started crawling other links right away. It took about 3 days for the meta content to show up for TheDogWay and it now ranks number 1 for ‘thedogway’ but just page one for ‘The Dog Way’.

Getting SEM Keyword Data From Apache Log Files

The use of adwords with the current form of The Dog Way is difficult because products are supplied through drop ship wholesalers that provide limited inventory and very small margins, but, on the plus side, no need to worry about fulfillment or inventory costs! Because the average gross product might be around $10, and I’m hoping the average margin per order will be around $20, there isn’t much room to bid. The CPCs I’m seeing right are $1.50. meaning a hopeful, break-even conversion rate of 7.5%, which is not the case – yet.

Lack of budget, lack of expected success, and a very limited product suite present some challenges. Couple that with a new domain, no relevant click-through-rate history, and the options of keywords I can profitably bid on became pretty limited.

I had initially planned on targeting very long tail phrases: add in the list of products, clean up the titles, and those titles became the keywords. That hit a wall when it became clear that inventory from Doba could change randomly and that Google would not enter keywords with insufficient search volume into the auction (which means the product names I had planned on targeting). Although, I imagine it’d still be possible to pick up those search queries through broad match somehow.

I did see a value on SEM though to generate a list of keywords to target for SEO,  and though there are other, cheaper ways to get these lists, which I’ll go over later, they don’t give an indication of conversion rates and usability stats.

I ended up making two very simple campaigns targeting ‘Dog Clothes’ and ‘Dog Coats’ and then the adgroups targeted the sub-categories. I put together about 5 keywords per ad group using broad match for each. I set a small budget and then ran it to see what would happen.

Of note:

-Most of the keywords did not end up showing, Google said the keywords were either too similar to other keywords, for example: ‘Affordable Dog Coats’ and ‘Affordable Dog Jackets’ and then that some were too low volume. Out of the roughly 50 keywords I entered, only 3 generated traffic. I wanted to see the actual search queries, but Google claimed there was insufficient volume to show those. This where grep comes in handy again.

I went to the access-logs file mentioned earlier and did a few other commands to specifically get the search queries. The command looked roughly like this, and then I’ll break it down.

grep -E “glcid|aclk” log_file.txt | awk ‘{print $11} | awk -Fq\= ‘{print $2}’ | awk -Fsource= ‘{print $1}’ | sed -e ‘s/%20/ /g’ | sed -e ‘s/”//g’ > ~/search_terms.txt

When reading anything that looks like code from me, please see my general disclaimer that basically says, “I’m a business guy, not a coder.”

The parts

grep -E “glcid|aclk” log_file.txt – Instructs the utility grep to look through the log file for instances of ‘glcid’ or aclk’, the parameters I’ve seen for adwords, and pulls out those lines.

The ‘|’, or pipe, is a way of outputting one commands output into another’s input through.

awk ‘{print $11}’ – awk is another unix utility for working with files, it’s very handy. Awk is very useful for looking at columns and uses whitespace as the column delimiter by default. The ‘{print $11}’ is a command to print just column 11, which, is the referring string, or what URL just sent the user to the landing page.

awk -Fq\= ‘{print $2} – The default column delimiter for awk is whitespace by default, but it’s possible to specify another delimiter using -Fpattern. In this case I used ‘q=’ because thats where Google puts the search query, but notice the ‘\’. The ‘\’ is a way to escape out a character and prevent it from being used as a special character. Once I specified the delimiter as ‘q=’, the referring string gets broken into two. The part I want, with the search query, is in column 2.

awk -Fsource= ‘{print $1}’ – Another use of the delimiter because there is still some part of the URL on the search query I don’t want. I basically used the same trick as a move to get down to just the search query.

sed -e ‘s/%20/ /g’ – Sed is a unix stream editor, another handy utility. The -e tells sed to edit the stream. The next part is essentially a find and replace. I am replacing the ‘%20’ with a ‘space’ to clean up the formatting. The ‘g’ at the end is a specification to make it global, or on all instances. The basic sed replace structure is this:

-sed ‘s/string_to_replace/new_string_to_enter/numberofinstancestoreplace’

sed -e ‘s/”//g’ – That gets rid of any quotation marks on the string, and we’re left with “ “.

> ~/search_terms.txt – Instead of piping the previous output to another command, we’re redirecting the output to a file. The ~/ specifies my home directory and then the file name is search_terms.txt.

And now we have a list of keywords looking like this:

benfica dog clothing

cheap pet clothes for small dog

cheap designer dog clothes

cheap designer dog clothes

dog clothes

clothyes for dogs



I’ll write another blog post about a couple of simple ways to clean up the file, you can you put it in excel to dedup, sort, count, et cetera.