Getting deep crawled

[chromate]

I'm being visited once a day or so by googlebot. Thing is, it only crawls the entrance page and has never been beyond that.

The site's kind of new, but it does have a temporary listing. I would have thought it would have been deep crawled by now considering the number of times googlebot's visited the site.

I'm not using any robots.txt file and my links are standard like:

http://www.mydomain.com/linkdir/

And I'm not using any meta tags.

I really want it to get deep crawled before the next update. Is there nothing I can do about it? Or are there any reasons beyond the robots.txt file that would prevent a deep crawl?

[Mayet]

Personally I would metatag the description at least..I know keyword metas are not as useful as they once were but i still use them just in case. I do't find they detract from the pages crawl value.
As to the first bit I read about this the other day. Apparently google does crawl the deep links, and you can find them if you google them but overall it takes your site as a whole and assumes the deeplink is exactly that..a deep link and wont return them highly in the search results.
From what I read and I will try and find a link to the article it is better to create subdomains and not to link them back to the main site. Then google sees them as separate entities. This way you can then create recipricol links around the board and boost each individual subject rankings. This is also easier for dmoz submission as they do not like listing deep links and have come down rather hard on it lately pruning many deeplinks from the directory.

I also submit every page singular. I sit there and just use the back button..submit..back button submit till i have every page submitted. I know google says you don't have to but I still found that google was only returning the url's I actually submitted highly on the results

Its also well worth to read on how google works. How they actually update through the datacentres. "The google dance"
It doesn't take place monthly as it did it's supposed to be live updating.

Once again this is my humble opinion only and only from what I have found myself asking the same questions.

[Chris]

Submitting subpages is definitely a good idea, one or two atleast, to get the ball rolling.

If your site hasn't been up very long though (two weeks) it wouldn't suprise me that Google has yet to deep crawl it.

[michael_gersitz]

Give it at least a month.

Also be sure to build a sitemap, so once it does "deep crawl" it will list your whole site.

[chromate]

I have submitted all my subpages once. I only have about 20 that I really want it to crawl. But still, googlebot never visits the other pages. Just the root page.

It's not really worth me building a site map because my structure is so simple really. The front page is like a complete site map in itself :)

I'll try submitting the "deeper" pages agian and see what happens.

[chromate]

I'm pleased to say, my site finally got deep crawled last night :)

[MarkB]

How can you tell? By stats, or by pages indexed on Google?

[chromate]

By looking at the stats and seeing what pages googlebot has crawled.

[chromate]

I just checked my stats again. GoogleBot's being going crazy crawling every possible page of my site. It's crawled 890 pages so far. Most of which I wasn't expecting would be crawled. More to the point, I didn't want them crawled because some of them have very minimal content. I just hope I don't get penalized for it! :eek:

[michael_gersitz]

How much is minumal content?

[chromate]

Couple of lines sometimes. I should have used a robots.txt file to exclude those ones. I didn't think google would follow their links due to the structure of the URL.

Oh well. I think it should be alright. Even if it's not, a quick email to them should sort it out.

[michael_gersitz]

I don't know how my new font site will do, because all the content is similar on most of the font pages, except for like 4 lines that decribe the fonts...

[Mayet]

Googles been acting weird I will say that.

Thanks for the font site. Thats great btw just what I was looking for.:D :D

[Mike]

I submitted to google about a week ago, and googlebot hasen't been seen crawling around yet:(

[chromate]

Do you mean you haven't been deep crawled or that you haven't been crawled at all? If the latter, try submitting again on a daily basis until you do get crawled. Wont do any harm.