e-httpxfer is a HTTP request transfer utility. This document describes how the uses of e-httpxfer, how it is installed, and how it is configured.
e-httpxfer is a utility which is used to catch HTTP request, to re-route those requests to a given (pre-defined) location, to collect all data there, and to send back the data to the originator.
The general working of e-httpxfer is explained below:
Using this approach, e-httpxfer is ideal as a handler on a public system, and can be used to handle requests to multiple "hidden" webservers, which themselves may be located e.g. in a LAN, unreachable by a direct connection.
e-httpxfer can be best described as a very light-weight and configurable proxy for HTTP traffic.
Througout this document, flags of e-httpxfer will be mentioned. The full explanation is given here.
The invocation of e-httpxfer is e-httpxfer flags. No non-flag
arguments are allowed. The supported flags are shown when e-httpxfer is
started with an argument, as in e-httpxfer x. The flags are also
discussed below.
-afile: Logs the activity to file. The activity is
shown as:
-cretries: Instructs e-httpxfer to compensate
retries times when trying to get a response from the webserver.
The default is (of course) 1 -- i.e., try just once.
You might want to use this flag when e-httpxfer is requesting data from a non-stable webserver, e.g., a webserver in a testing environment, which is re-started often. Restarting the webserver would otherwise break the request transmission and e-httpxfer would report a read error.
-dport: Instructs e-httpxfer to "daemonize" while
binding to the specified port. The default is that e-httpxfer will
run once (single-shot) and will use stdin and stdout as its
inputs.
-rresource: Instructs e-httpxfer to load resouce
as its resource file, instead of /usr/e/etc/e-httpxfer.rc. (The
resource -or configuration- file is further discussed below.)
-sskip,skip,skip,...: Instructs e-httpxfer to
skip the indicated HTTP header lines. This means that when the lines
are seen in the request, the lines are not sent to the true
webserver. For example, "smarter" browsers will try to determine
whether or not to request a page, by supplying a header line that
starts with If-Modified-Since. The webserver is supposed to send
the requested information only if it was modified after the given
date. By suppressing this line, the webserver will always send the
requested information.
-ttimeout: By default, e-httpxfer uses a timeout of
300 seconds (5 minutes). If the true webserver doesn't respond by
then, then e-httpxfer assumes that the request cannot be
fulfilled. The flag -t defines this timeout in seconds.
-v: This makes the run more "verbose", e-httpxfer logs
its actions in /tmp/e-httpxfer.log. Enabling logging is a good way
to see what header lines are received and how e-httpxfer handles
them.
-V: This instructs e-httpxfer to show its version ID and
to stop.
To build e-httpxfer, the following e-tunity packages are necessary:
Once the support packages e-lib, epp-lib and misc are available and installed, e-httpxfer compiles and installs out-of-the-box.
Configuring and using e-httpxfer is a two-step process: first, e-httpxfer is bound to the HTTP port of a server (port 80). Second, its internal configuration file must be adapted.
e-httpxfer knows two operating modes: an inetd-based single shot mode, and a server (daemon) mode.
4.1.1: Configuring the inetd mode
To enable e-httpxfer as a "service" at port 80, two files must be changed
by the superuser (root): /etc/services and /etc/inetd.conf. The
changes in /etc/services are:
# service port/type
httpxfer 80/tcp
Note that other predefined services for port 80, type tcp, may co-exist as far
as the file /etc/services is concerned.
The changes in /etc/inetd.conf are:
# service net type state.max user path to binary
httpxfer stream tcp nowait.1000 root /usr/e/bin/e-httpxfer e-httpxfer
Note that in inetd.conf, only one service for port 80 may be present.
E.g., if services also defines a service www, for port 80, then the
line staring with www in /etc/inetd.conf must be either deleted or
commented out.
Once the files /etc/services and /etc/inetd.conf are changed, the
daemon inetd must be restarted, typically using killall -1 inetd.
4.1.2: Configuring the daemon mode
The startup of e-httpxfer as daemon only requires the presence of a flag
-dport, thereby instructing e-httpxfer to bind to the specified
port. Often a start script in or under /etc/rc.d/ will be necessary.
e-httpxfer's configuration file by default resides in
/usr/e/etc/e-httpxfer.rc, though a different path may be specified using
the switch "-r alternate-file". If needed, this switch must be
stated in /etc/inetd.conf or in e-httpxfer's start script.
The configuration file is a list of two- or three column entries. The first column specifies a host setting received HTTP request. When e-httpxfer matches this host setting, then the second column entry is taken as the webserver to connect to. When present, a third column specifies options to e-httprequest that apply to only this host entry.
An example is given below:
# Requests for... should go to... using optional flags
e-tunity.com 192.168.1.50
www.e-tunity.com 192.168.1.50
e-tunity.com:81 192.168.1.50:81
e-tunity.net 192.168.1.100 /usr/e/bin/e-httpxfer \
e-httpxfer -v -a /tmp/act-net.log
www.e-tunity.net 192.168.1.100 /usr/e/bin/e-httpxfer \
e-httpxfer -v -a /tmp/act-net.log
e-tunity.nl 192.168.1.100:81
www.e-tunity.nl 192.168.1.100:81
default 192.168.1.80
The sample entries are:
e-tunity.com and www.e-tunity.com are
handled via the webserver at 192.168.1.50. Both host
specifications do not specify a port, so that the default HTTP port 80
applies. In this example, 192.168.1.50 would typically be a
webserver in a private LAN.
e-tunity.com:81 also go to 192.168.1.50, but
must be handled there via port 81. Note that the configuration file
does not provide for www.e-tunity.com:81.
e-tunity.net (or when prefixed with www), go
to 192.168.1.100. These entries also supply a third column, which
is formulated a-la the /etc/inetd.conf entry. In these two entries
the flags -v and -a /tmp/act-net.log are supplied. The flag
-v makes e-httpxfer more "verbose" and logs actions in
/tmp/e-httpxfer.log; the flag -a must be followed by a file
name, and instructs e-httpxfer to log its activity in the given
file.
This entry also shows how "long" lines can be wrapped using \-continuation.
e-tunity.nl (or when prefixed with www) go
to the same webserver as the requests for e-tunity.net, but to
port 81 instead of the default port.
192.168.1.80.
This example of course assumes that the requests to e-tunity.com,
e-tunity.net etc., all arrive at the server where e-httpxfer is
installed. This is a DNS-issue and is not further discussed here.
As far as the performance is concerned, the following rules of thumb apply.
If you still suspect that the "single-shot" approach is a bottleneck, please switch to daemon mode.
www.altavista.com will be significantly slower than having them
handled by 209.73.164.92. Note again that each HTML page will
typically contain multiple elements, all pointing to the same host
(such as /images/whatever.gif, /css/sheet.css, framed pages).
That means that e-httpxfer will be re-started many times for just
one page: having the handler address as an IP number helps a lot.
/tmp/e-httpxfer.log and will ensure that the log doesn't clog the
disk: the log is "rotated" when it reaches 100K, and no more than 5
rotated logs are kept around. However, this also means that every run
of e-httpxfer may involve log rotation, compression etc..
Therefore, once e-httpxfer is tested, the flag -v in
the activation command should be removed.