如何在AEM / CQ中添加robots.txt文件？

问题描述

如何将robots.txt文件添加到AEM服务器以为AEM中的Web爬网程序提供规则？

How to add the robots.txt file to an AEM server to provide rules for Web Crawlers in AEM?

推荐答案

大多数您将引用
实现此目的。

Most you will refer to This LinkTo implement this.

尽管它似乎达到了目的，但您会注意到一件事可能有点不正确。

Although it may seem to serve the purpose you will notice one thing that could be a little "not right".

直接在crxde中添加robots.txt文件会导致在根级别创建nt：file类型的节点。

因此，当您点击而不是您会在屏幕/浏览器上显示文件下载。

So when you hit http://localhost:4502/robots.txt instead of you displaying on the screen/browser the file downloads.

这是因为默认的GET servlet 。 servlet标识节点类型为nt：file，并发送内容类型为

This is because of the Default GET servlet. The servlet identifies that the node type is nt:file and sends response with the content type as

Content-Type: application/octet-stream
Content-Disposition: attachment;filename=robots.txt

要克服此工具过滤器如下。这样，您将跳过对Sling的Default GET Servlet的调用，并能够提供您自己的内容类型。

To overcome this implement the filter as follows. By doing this you will skip call to the Default GET Servlet of Sling and will be able to provide a content type of your own.

 package com.hds.exp.filters;

import org.apache.felix.scr.annotations.sling.SlingFilter;

import java.io.IOException;
import java.io.PrintWriter;

import javax.servlet.FilterChain;
import javax.servlet.FilterConfig;
import javax.servlet.ServletException;
import javax.servlet.ServletRequest;
import javax.servlet.ServletResponse;
import javax.servlet.http.HttpServletRequest;

import org.apache.felix.scr.annotations.Properties;
import org.apache.felix.scr.annotations.Property;

@SlingFilter(order=1)
@Properties({
    @Property(name="service.pid", value="com.hds.exp.filters.RobotsFilter",propertyPrivate=false),
    @Property(name="service.description",value="Provides Robots.txt", propertyPrivate=false),
    @Property(name="service.vendor",value="DD Exp", propertyPrivate=false),
    @Property(name="pattern",value="/.*", propertyPrivate=false)
})
public class RobotsFilter implements javax.servlet.Filter{

    @Override
    public void destroy() {
        // Unused
    }

    @Override
    public void doFilter(ServletRequest request, ServletResponse response, FilterChain chain)
            throws IOException, ServletException {

        HttpServletRequest httpServletRequest =(HttpServletRequest) request;
        if(httpServletRequest.getRequestURI().equals("/robots.txt"))
        {
            response.setContentType("text/plain");
            PrintWriter writer = response.getWriter();
            writer.print("User-agent: *");
            writer.print("\n");
            writer.print("Disallow: /");
            writer.print("\n");
            writer.flush();
        }
        else
        {
            chain.doFilter(request, response);
        }
    }

    @Override
    public void init(FilterConfig arg0) throws ServletException {
        // Unused
    }

}

这篇关于如何在AEM / CQ中添加robots.txt文件？的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持！