分布式架构——第12篇:tomcat、memcache分布式session

memcached-session-manager是一个开源的高可用的tomcat session共享解决方案。session可以先放在服务器本地,等请求处理完成之后再同步到后端的memcached服务器。这样,当Web Server宕机的时候,其他的Web Server可以从memcached服务器中还原出session状态。

构建tomcat、memcache集群

这里使用docker技术,构建集群。推荐先阅读 分布式架构——第9篇:MACVLAN网络(Docker直连物理网络)。 1. 构建memcached集群 直接使用官方提供的memcached容器,想了解更多,可以阅读 https://docs.docker.com/samples/library/memcached/

$ sudo docker run --net=pub_net --ip=192.168.1.16 --name macvlan_mc1 --restart always -d memcached
$ sudo docker run --net=pub_net --ip=192.168.1.17 --name macvlan_mc2 --restart always -d memcached
$ sudo docker run --net=pub_net --ip=192.168.1.18 --name macvlan_mc3 --restart always -d memcached

  1. 构建tomcat集群 这里使用的是Apache Tomcat/8.5.34,关于docker tomcat的更多内容 https://docs.docker.com/samples/library/tomcat/
    $ sudo docker run --net=pub_net --ip=192.168.1.19 --name macvlan_tc1 --restart always -d tomcat
    $ sudo docker run --net=pub_net --ip=192.168.1.20 --name macvlan_tc2 --restart always -d tomcat
    $ sudo docker run --net=pub_net --ip=192.168.1.21 --name macvlan_tc3 --restart always -d tomcat
  • JAR包依赖 关于tomcat、memcached、kryo请先阅读memcached-session-manager,文中提到了JAR包依赖,kryo-serializer: msm-kryo-serializer, kryo-serializers-0.34+, kryo-3.x, minlog, reflectasm, asm-5.x, objenesis-2.x ... 这里下载好后,通过sshssh配置以及远程文件传输上传到服务器。再通过docker cp命令将所有JAR包拷贝至tomcat/usr/local/tomcat/lib/目录中。 e.g.

    $ sudo docker cp memcached-session-manager-2.3.0.jar macvlan_tc1:/usr/local/tomcat/lib/

    相关下载路径: [1] http://repo1.maven.org/maven2/de/javakaffee/msm/memcached-session-manager/2.3.0/memcached-session-manager-2.3.0.jar [2] http://repo1.maven.org/maven2/de/javakaffee/msm/memcached-session-manager-tc8/2.3.0/memcached-session-manager-tc8-2.3.0.jar [3] http://repo1.maven.org/maven2/de/javakaffee/msm/msm-kryo-serializer/2.3.0/msm-kryo-serializer-2.3.0.jar [4] http://repo1.maven.org/maven2/de/javakaffee/kryo-serializers/0.42/kryo-serializers-0.42.jar [5] http://repo1.maven.org/maven2/com/esotericsoftware/kryo/4.0.2/kryo-4.0.2.jar [6] http://repo1.maven.org/maven2/com/esotericsoftware/minlog/1.3.0/minlog-1.3.0.jar [7] http://repo1.maven.org/maven2/com/esotericsoftware/reflectasm/1.11.7/reflectasm-1.11.7.jar [8] http://repo1.maven.org/maven2/org/ow2/asm/asm/6.2.1/asm-6.2.1.jar [9] http://repo1.maven.org/maven2/org/objenesis/objenesis/3.0.1/objenesis-3.0.1.jar [10] http://repo1.maven.org/maven2/net/spy/spymemcached/2.12.3/spymemcached-2.12.3.jar

  • 配置content.xml 默认容器中没有安装vivim工具,可以使用apt updateapt install vim进行安装。

    $ sudo docker exec -it macvlan_tc1 /bin/bash
    $ vi /usr/local/tomcat/conf/context.xml

    <?xml version="1.0" encoding="UTF-8"?>
    <!-- ... -->
    <Context>
    <!-- ... -->
    <Manager className="de.javakaffee.web.msm.MemcachedBackupSessionManager"
    memcachedNodes="n1:192.168.1.16:11211,n2:192.168.1.17:11211,n3:192.168.1.18:11211"
    lockingMode="auto"
    sticky="false"
    requestUriIgnorePattern= ".*\.(png|gif|jpg|css|js)$"
    sessionBackupAsync= "false"
    transcoderFactoryClass="de.javakaffee.web.msm.serializer.kryo.KryoTranscoderFactory"/>

    </Context>

    番外1, org.apache.catalina.LifecycleException异常,一般是由于JAR包依赖没有到位,或者版本有误,我是asm-6.2.1.jarminlog-1.3.0.jarobjenesis-3.0.1.jarreflectasm-1.11.7.jar几个包没拷贝进去就报错了。

    25-Oct-2018 09:24:14.032 INFO [localhost-startStop-1] de.javakaffee.web.msm.MemcachedSessionService.startInternal  starts initialization... 
    (configured nodes definition n1:192.168.1.16:11211, n2:192.168.1.17:11211, n3:192.168.1.18:11211, failover nodes null)
    25-Oct-2018 09:24:14.032 SEVERE [localhost-startStop-1] org.apache.catalina.core.StandardContext.startInternal The session manager failed to start
    org.apache.catalina.LifecycleException: Failed to start component [de.javakaffee.web.msm.MemcachedBackupSessionManager[/examples]]
    at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:167)
    at org.apache.catalina.core.StandardContext.startInternal(StandardContext.java:5273)
    ...

    番外2, java.lang.IllegalArgumentException异常,这个是由于memcachedNodes编写有误,我原来是这样写的memcachedNodes="n1:192.168.1.16:11211, n2:192.168.1.17:11211, n3:192.168.1.18:11211"报错了。后来去掉逗号后面的空格就好了。

    Caused by: java.lang.IllegalArgumentException: Configured memcachedNodes attribute has wrong format, 
    must match ([\w]+):([^:]+):([\d]+)(?:(?:\s+|,)([\w]+):([^:]+):([\d]+))*
    at de.javakaffee.web.msm.MemcachedNodesManager.createFor(MemcachedNodesManager.java:197)
    at de.javakaffee.web.msm.MemcachedSessionService.createMemcachedNodesManager(MemcachedSessionService.java:495)
    ...

分布式session测试

Tomcat有个内置的session页面${TOMCAT_HOME}/examples/servlets/servlet/SessionExample,直接访问该页面即可看到相关的session内容。 1. Web Server 1

http://192.168.1.19:8080/examples/servlets/servlet/SessionExample

Session ID: 9DFBEB4FF0562B67C75889B37E9E9A4C-n2
Created: Thu Oct 25 10:16:50 UTC 2018
Last Accessed: Thu Oct 25 10:16:50 UTC 2018

  1. Web Server 2

    http://192.168.1.20:8080/examples/servlets/servlet/SessionExample

    Session ID: 02A9F4784F6BC4BBF8E23DF72B5D0622-n3
    Created: Thu Oct 25 10:17:25 UTC 2018
    Last Accessed: Thu Oct 25 10:17:28 UTC 2018

  2. Web Server 3

    http://192.168.1.21:8080/examples/servlets/servlet/SessionExample

    Session ID: 532EEDA7F107E08E8A44C39E70E7FE0F-n3
    Created: Thu Oct 25 10:17:51 UTC 2018
    Last Accessed: Thu Oct 25 10:17:55 UTC 2018

模拟服务器宕机

假设,这时候Tomcat服务器宕机了。这里强制重启docker容器,模拟服务器宕机。

$ sudo docker restart macvlan_tc3
macvlan_tc3
$ sudo docker restart macvlan_tc2
macvlan_tc2
$ sudo docker restart macvlan_tc1
macvlan_tc1

session恢复验证

OK,启动完毕之后,再次发起Web Server请求... 结果显示,session仍然与原来的保持一致(已从memcache中恢复)。 e.g.

http://192.168.1.21:8080/examples/servlets/servlet/SessionExample

Session ID: 532EEDA7F107E08E8A44C39E70E7FE0F-n3
Created: Thu Oct 25 10:17:51 UTC 2018
Last Accessed: Thu Oct 25 10:19:10 UTC 2018

memcache相关查询

网上有一篇关于查询memcache内容的文章https://www.darkcoding.net/software/memcached-list-all-keys/。另外还有一篇谈memcache内存管理机制的文章https://www.zybuluo.com/phper/note/443547下面摘录了几段:在熟悉memcached的内存管理之前,我们先拿小学生的格子作业本来举例子,把格子本比作memcached的内存分配。每个作业本(memcached的内存空间),它都有很页(slab),每一页里面有很多方格子(chunk),每个格子里面可以写字(item)。 - slab、chunk 上面说到的格子本的一页纸,就一个是slab, 是memcached分配的一块内存空间,默认大小为1M。memcached会将内存空间分配成一个一个的slab,还会把一个slab分割成一个一个的格子,也就是一个一个chunk,比如说1M的slab分成两个0.5M的chunk,slab和chunk其实都是代表实质的内存空间,chunk是slab分割后的更小的单元。所以:slab就相当于作业本中的“一页纸”,而chunk则是把这一页纸中的一个个的“格子”。 - item item是我们要保存的数据,也就是我们需要在“格子”中写入的“字”。 - slabclass 通过上面我们知道,slab(都假设为1M)会割成一个个chunk,而item往chunk中塞。那么问题来了:我们要把这个1M的slab割成多少个chunk?就是一页纸,要画多少个格子?我们往chunk中塞item的时候,item总不可能会与chunk的大小完全匹配吧,chunk太小塞不下或者chunk太大浪费了怎么办?就是我们写字的时候,格子太小,字出界了,或者我们的字很小写在一个大格子里面好浪费。所以memcached的设计是,我们会准备“几种不同格子的slab”,也就是说根据“slab分割的chunk的大小不一样”来分成“不同的种类的slab”,而 slabclass就是“slab的种类”的意思了。

以下是一个对memcache的JAVA查询程序:

package org.nt.memcached;

import java.util.Map;

import com.whalin.MemCached.MemCachedClient;
import com.whalin.MemCached.SockIOPool;

public class TomcatSession {
public static void main(String[] args) {
String[] servers = {
"192.168.1.16:11211","192.168.1.17:11211","192.168.1.18:11211"
};
SockIOPool pool = SockIOPool.getInstance();
pool.setServers(servers);
pool.setFailover(true);
pool.setInitConn(10);
pool.setMinConn(5);
pool.setMaxConn(25);
pool.setMaintSleep(30);
pool.setNagle(false);
pool.setSocketTO(3000);
pool.setAliveCheck(true);
pool.setHashingAlg(SockIOPool.CONSISTENT_HASH);
pool.initialize();
MemCachedClient mcc = new MemCachedClient();

Map<String, Map<String, String>> statsMap = mcc.stats();
System.out.println(statsMap);

Map<String, Map<String, String>> statsItemsMap = mcc.statsItems();
System.out.println(statsItemsMap);

Map<String, Map<String, String>> statsCacheDumpMap = mcc.statsCacheDump(3, 0);
System.out.println("\n"+statsCacheDumpMap+"\n");

Object obj = mcc.get("9DFBEB4FF0562B67C75889B37E9E9A4C-n2");
System.out.println(obj);
}
}

部分查询结果摘录:

...
{192.168.1.16:11211={bak:validity:02A9F4784F6BC4BBF8E23DF72B5D0622-n3=[20 b; 1540468300 s]
, bak:validity:532EEDA7F107E08E8A44C39E70E7FE0F-n3=[20 b; 1540468298 s]
}, 192.168.1.18:11211={validity:02A9F4784F6BC4BBF8E23DF72B5D0622-n3=[20 b; 1540468300 s]
, validity:532EEDA7F107E08E8A44C39E70E7FE0F-n3=[20 b; 1540468298 s]
, bak:validity:9DFBEB4FF0562B67C75889B37E9E9A4C-n2=[20 b; 1540468448 s]
}, 192.168.1.17:11211={validity:9DFBEB4FF0562B67C75889B37E9E9A4C-n2=[20 b; 1540468449 s]
}}
...

References: [1] https://www.cnblogs.com/whgk/p/6422391.html [2] https://www.cnblogs.com/notDog/p/5341219.html [3] https://code.google.com/archive/p/memcached-session-manager/downloads [4] https://blog.csdn.net/dongdong9223/article/details/71425077 [5] http://www.importnew.com/19133.html [6] https://github.com/magro/memcached-session-manager/wiki/SetupAndConfiguration#example-for-non-sticky-sessions--kryo [7] https://www.darkcoding.net/software/memcached-list-all-keys/ [8] https://www.zybuluo.com/phper/note/443547 [9] http://www.cnblogs.com/yinrq/p/5019024.html [10] https://docs.docker.com/samples/library/tomcat/ [11] https://docs.docker.com/samples/library/memcached/ [12] 大型分布式网站架构设计与实践.陈康贤著