前言

游戏服务端开发是一个非常神秘的软件开发工种，作为游戏开发的两大子类之一，其关注度远低于游戏客户端开发。常见的游戏开发书籍《游戏引擎架构》、《游戏编程精粹》等基本都是介绍客户端相关的渲染、物理、动画等内容。Bilibili上的知名游戏开发经验分享账号Games Webinar上传的五百多个视频里关于游戏服务端的一只手都数的过来，都集中在Games 104里。相对于汗牛充栋的游戏客户端开发书籍，关于游戏服务端开发的书籍可谓屈指可数，比较知名的只有《网络多人游戏架构与编程》。这些游戏服务端开发相关书籍大多在花一半以上篇幅来介绍网络相关的内容：如何使用基础socket来做网络通信，如何降低延迟，服务器与客户端之间的互联拓扑。由于这些内容与常规的互联网服务端开发重合度太高，同时又缺少类似于分布式一致性这类高端内容的介绍，给了读者一种游戏服务端开发没啥技术含量的错觉，以至于知乎上有这么几个提问:互联网服务端技术领先游戏服务端几十年？; 游戏服务端技术/人员整体水平是不是已经落后于互联网服务端?。

正所谓"指法无优劣，功力有高下"，游戏服务端与互联网服务端两者要处理的业务有着很大的差异，无法直接对比技术栈。在上面的知乎问题的回答里，很多人都提到了这两者最大的差别：互联网服务是短连接、无状态、高延迟的，而游戏服务是长连接、有状态、低延迟的。但是这种过于干练的概括对于理解游戏服务端并没有很大的帮助。刚好那段时期本人用Python做了太多的游戏节日活动，有点烦躁，想通过自己从零搭建一些游戏服务端通用的组件来更好的理解隐藏在脚本接口之下的游戏服务端的核心业务。在看到这个知乎问题之后，萌生出了一个大胆的想法：用cpp做一个Demo性质的游戏服务端引擎mosaic_game来复刻当前使用的自研闭源游戏服务端引擎的核心功能，同时将这个引擎的功能设计与实现归纳整理为一本完整的书籍，为那些对游戏服务端有兴趣的开发人员提供一份比较系统且充满细节的参考资料。希望通过本书的介绍，能够让大家对游戏服务端开发有一个更全面的认识，了解其功能集合与职责所在。

随着后续工作的变动，逐渐开始接触到其他的商用游戏服务端引擎，最知名的就是BigWorld和Unreal Engine。我所在的大世界服务端组的业务重心就是将BigWorld核心的无缝大世界功能移植到Unreal Engine上，以提升Unreal Engine的单场景玩家承载能力。在做这个无缝大世界功能移植的过程中需要反复的去查阅这两个引擎的源代码，再加上我自己闭门造车的mosaic_game，发现虽然这三个游戏服务端引擎面对的问题相似，但是对应的解法却各有不同。同一个问题三种解法，着实有趣。

兴致起来了之后，我便修改了原有的开发计划，给mosaic_game增加了无缝大世界的支持。同时本书的内容也进行了大量的扩充，同一个业务主题会多个章节来分别介绍这三个引擎的解决方案，大致覆盖了下述内容：

• 通信管理，包括客户端与服务器之间，服务器与服务器之间的通信
• 会话管理，包括账号创建、登录、顶号、下线、断线重连等
• 数据持久化，包括玩家数据、场景数据、社交数据等
• 客户端请求响应，包括移动、聊天、社交、战斗、场景切换等
• 玩法流程控制，包括副本流程、怪物AI等
• 实体状态同步，包括实体的位置同步、属性同步等，
• 分布式场景管理，包括分布式场景的动态扩缩容、无缝迁移等

在介绍这这些内容的时候，我都会附加这些内容的核心代码来方便读者去了解这些内容的实现细节。在章节膨胀和代码填充这两个因素的影响下，本书的篇幅也从原来的600页不到扩充到了现在的1500页。太多的代码贴在书中，会导致书籍的可读性下降。如果读者不想深究实现细节的话可以对这些代码进行跳过，对于不感兴趣的章节也可以跳过。

由于本人经验和精力有限，上述内容之外的一些游戏服务端的非常重要的业务并没有在本书中展开介绍，例如技能与战斗系统、脚本系统、日志收集系统、性能监控系统、运维部署系统、热更新系统、容灾系统等，欢迎对这些内容有深刻理解的游戏服务端开发人员来对本书进行补充。

游戏服务端架构介绍

游戏服务端架构演进

其实关于游戏服务端架构演进的内容，已经有很多前辈总结过了。本人从业没多少年，经历过的项目也是屈指可数，所以本章的存在只是为了全书的完整性，简单的介绍一下。对于更多细节感兴趣的读者可以去阅读skywind3000前辈写的游戏服务端架构发展史。

单进程架构

单进程架构是游戏服务端最简单也最久远的架构，起源于1978年诞生的游戏服务器Multi-User Dungeon，简称为MUD1。此时游戏服务器只有一个进程，玩家的客户端可以通过telnet协议连接到游戏服务器，然后就可以通过文本输入来控制自己的角色来进行游戏。

在MUD1中，服务器进程只有一个线程，网络收发使用非阻塞的网络IO模型，同时线程里会每隔1秒钟更新一次所有对象（网络收发，更新对象状态机，处理超时，刷新地图，刷新NPC）。同时玩家数据的持久化使用的是本地文件系统，每个玩家的数据都存储在一个单独的文件中。

数据库进程

随着互联网和硬件的发展，游戏内容也从纯文字进化为了图形化。由于图形化界面相对于纯文本界面能携带更多的信息，就使得单个玩家的游戏数据量急速膨胀。同时由于网络的逐渐普及，游戏服务器要服务器的玩家数量也逐渐增多。在这两个因素的影响下，基于小文件的持久化系统已经力不从心，因此开始引入数据库作为新的持久化手段。

同时为了规范化游戏内对数据库的访问，开始引入数据库访问层。数据库访问层负责处理所有对数据库的访问请求，包括玩家数据的读取和写入。

场景进程

当玩家在线数量持续增多时，当前的单场景进程的CPU就成为了瓶颈。为了提高游戏服务器的同时在线人数，开始对游戏场景进行水平划分，每个场景进程负责若干个独立的场景，每个场景负责管理一部分玩家。这种结构也就是常说的分区分服。

在这样的设计下，只要有无限的场景进程实例，就可以无限的扩展游戏服务器的同时在线人数。

由于玩家不会一直呆在同一个场景里，但是不同的场景被不同的场景进程负责。此时最简单的方法就是从原有的场景进程中下线并断开连接，然后往新的场景进程发起登录请求，这个下线再上线的过程会涉及到非常多的状态修改和数据交互。为了避免这些操作对游戏的影响，不同的场景进程之间开始出现互连，这样玩家数据就可以从一个场景进程传递到另外一个场景进程，避免了存库再从数据库中进行拉取的操作，这个过程就叫做玩家的迁移(Migration)。此时客户端只需要在迁移完成之后根据角色所在的最新场景进程通信地址发起一个新连接，同时断开对老的场景进程的连接，即可恢复游玩。这样就避免了繁重的上下线流程，减少了切换场景进程的延迟。

网关进程

由于常见的网络协议都建立在TCP之上，而TCP连接的握手和关闭过程都比较冗长，这就给玩家切换场景进程带来了不少的延迟。为了进一步减少这种迁移延迟，开始引入网关进程。网关进程承担的角色与Nginx类似，作为反向代理中转场景进程与客户端之间的通信，因此网关进程会与所有的场景进程建立连接。

此时客户端不再连接场景进程，只连接网关进程。这样做的好处是当一个玩家切换场景服务器的时候，客户端连接是不会断的，只需要在网关进程上更新一下这个客户端对应的场景进程关联。同时由于网关进程的功能比较简单，基本只负责通信转发，所以其单进程承载量可以达到上万。网关进程的引入还有一个好处就是可以隐藏后面的场景进程，提供一个统一的接入点的同时，避免了内部服务器暴露在公网，从而减少了服务器被DDOS和入侵的风险。

服务进程

随着游戏社交性的增强，游戏内玩家的交互对象不再仅限于同场景的其他玩家，还可以与其他场景的玩家进行交互，典型例子就是在副本内与其他不在当前副本里的好友聊天。为了支持这样的与场景无关的玩法需求，开始引入服务进程。服务进程负责处理所有与场景无关的游戏逻辑，比如登录服务、场景管理服务、玩家之间好友、组队、聊天等。

此时所有的场景进程都与服务进程建立连接，服务进程也会与其他的所有服务进程发起连接，因为服务之间可能出现相互调用，例如给好友发消息会先通过好友服务验证两者之间的好友关系是否存在，然后再通过聊天服务将聊天记录存库，最后再通过在线通知服务来提醒对方进行消息接收。

消息总线

在上面的服务器架构图中可以发现，场景进程与服务进程在某种程度上来说是等价的，对于包含所有场景进程的集合A和包含所有服务进程的集合B而言，A中的任意进程都需要维持到A+B这个集合中任意进程的连接，同时B中的任意进程也需要维持到A+B这个集合中任意进程的连接。所以A+B集合内的连接数量是$N^2$的，是集合总大小的平方复杂度。虽然服务进程是有限的，但是场景进程基本上可以水平的无限扩充，同时场景进程还可以动态的扩缩容。在这种情况下，随着场景进程的数量逐渐增多，连接管理也会越来越复杂。为了减少连接管理的复杂度，隐藏进程间通信的细节，开始引入消息总线进程。

所有的场景进程、服务进程、网关进程、数据库代理进程都与消息总线建立连接，并通过消息总线来进行通信。这样就避免了每个进程都需要维护到其他所有进程的连接，同时也隐藏了进程间通信的细节，简化了开发和运维的复杂度。

一般来说，消息总线相关的进程会在每台物理机器上部署一个，这台物理机上的所有其他进程都会发起到本机消息总线进程的连接。然后每台机器上的消息总线进程又会统一的连接到一个全局的消息总线分发器，这个分发器会负责将消息路由到对应机器的消息总线进程上，并由目标机器的消息总线进程进一步分发到业务进程中。这样设计相对于所有业务进程都直接连接到全局的消息总线分发器来说，能够有效的减少消息总线分发器的网络连接数量。虽然消息转发相对于直接通过消息总线分发器来转发多了两次通信，但是这两次通信都是本机通信，速度很快，因此不会带来显著的延迟。如果消息的目标在本机的话，则不需要经过消息总线分发器，直接可以通过本机的消息总线进程来直接投递。

缓存服务

在当前设计下，客户端对于游戏数据的读取都需要经过网关->场景进程这两层中转，如果数据在服务进程上则会多加一层。对于一些变化频率不高但是拉取次数很多的数据，例如玩家的头像基本信息、定期结算的排行榜等，这些数据可以考虑放到游戏服务器之外去存储。这样客户端对这些数据的查询就可以完全不走前述的网关->场景进程这两层中转，同时有效的降低游戏服务器的CPU负载和流量压力。典型的数据缓存服务就是Redis集群，这个集群通过http的方式来暴露数据的查询与修改接口，这样客户端与服务端都可以很方便的与这个集群执行互操作。同时由于Redis集群的性能与吞吐量都很高，整体的机器收益是正的。而且这个Redis集群一般非常稳定，不需要跟随游戏服务器去更新部署，不会给服务端带来额外的维护负担。综合上述几个优点，基于Redis的数据缓存服务在游戏服务端非常流行。

分布式场景

在上述架构中，一个场景只会存留在一个场景进程中，所以这个场景内的玩家数量就会受到单个进程的计算能力限制。为了提升单场景的玩家数量上限，场景进程开始引入多线程，将一些计算密集型的阻塞任务放到多线程中执行，例如寻路线程、物理线程、网络线程、AOI线程等。这样做的好处是可以充分利用多核处理器的性能，降低主线程的一些负载，从而可以提升主线程单场景的玩家数量。但是由于游戏内的绝大部分的业务逻辑都是线性的，不方便拆分为异步任务投递到其他线程中，所以主线程单场景的玩家数量提升终归有限。

为了绕过这种由于主线程的性能瓶颈导致的单场景玩家数量上限问题，开始引入分布式场景。分布式场景的设计思想是将一个场景拆分成多个子场景，每个子场景都运行在一个独立的场景进程中。这样做的好处是可以将场景内的玩家数量分散到多个进程中，从而可以极大提升单逻辑场景的玩家数量上限。

这个场景划分并不是预先指定好的，而是随着场景内玩家数量的变化而动态调整的。例如当场景内玩家数量超过了某个阈值时，就会将场景拆分成多个子场景。当场景内玩家数量减少到某个阈值时，就会将多个子场景合并成一个场景。这个动态调整子场景布局的概念就叫做动态分区(Dynamic Partitioning)。为了方便计算边界以及合并、拆分，这些子场景的形状一般都是矩形。

由于玩家能够自由的在不同的子场景之间移动，一旦玩家从一个子场景移动到另一个子场景，就需要将玩家从原场景进程中移除，并将玩家添加到目标场景进程中，也就是需要执行一次玩家的迁移(migration)流程。由于常规的状态同步设计里一个玩家只能看到单进程内的其他玩家，这种越过边界导致的进程切换就会让客户端出现非常明显的同步状态变化。即属于上一个场景的实体都会被销毁，同时会突然同步当前场景里可见的一些实体。这种突然且剧烈的状态变化会给游戏体验带来非常严重的割裂感，解决这种由于迁移导致的客户端同步状态剧烈变化的技术就是无缝迁移(Seamless Migration)。

游戏服务端架构样例

在本书中，我将详细的介绍Bigworld、Unreal Engine、mosaic_game这三个游戏引擎在服务端的一些业务上的具体实现。为了方面读者理解后面章节中对那些充满了源代码的内容，在这里就先大致的介绍一些这三个引擎的基础设计以及一些核心概念。

BigWorld 服务端介绍

目前可以参考的服务端无缝大世界的资料只有BigWorld游戏引擎，使用这个引擎的游戏少之又少，其中最为知名的是坦克世界WorldOfTanks。根据其官方ppt6-years-of-bigworld-engine-evolution-caused-by-the-success-of-world-of-tanks，上面清楚的写着WorldOfTanks使用BigWorld引擎达到了一百万的ConCurrent User：

虽然使用这个引擎的游戏不多，幸运的是目前在Github上有这个引擎的开源代码，三年前公开在https://github.com/v2v3v4/BigWorld-Engine-14.4.1，我们可以通过这个公开源代码的版本来探究整体的服务端架构设计以及实现细节。下图就是bigworld服务端的整体架构:

在这个架构图里，我们可以看到bigworld服务端主要由以下几种进程角色组成：

• CellApp：场景服务器进程，每个进程负责管理一个或多个Cell。这里的每个Cell都对应一个分布式场景的矩形子区域。
• CellAppMgr：全局唯一的CellApp管理进程，负责维护所有CellApp的状态以及Cell的创建、销毁、迁移等操作。
• BaseApp: 实体服务进程，所有的可迁移实体都首先在BaseApp上建立，然后再根据其场景与位置信息往对应的CellApp上创建一个副本。同时对于玩家这种实体，BaseApp还会负责中转CellApp上的实体与对应客户端之间的消息交互。
• BaseAppMgr：全局唯一的BaseApp管理进程，负责维护所有BaseApp的状态以及BaseApp的创建、销毁、迁移等操作。
• LoginApp：登录服务器进程，负责处理玩家的登录、注册、验证等操作。
• DBMgr：全局唯一的数据库服务进程，负责代理所有对数据库的请求，。目前bigworld中使用的数据库是MySQL，但是DBMgr的设计是可以扩展到其他数据库的。

此外还有两个非常重要的进程角色没有在图中体现：一个叫做BWMachined进程，这个进程每个物理机上运行一个，其作用相当于消息总线和运维管理工具；另外一个叫做Reviver进程，这个全局只有一个，负责上述进程的心跳维持和崩溃时的重启。

在上述进程角色中，LoginApp和BaseApp是客户端可以直接连接的，其他进程则不允许客户端连接。因此官方说明书推荐LoginApp和BaseApp所在的物理机配置两个网卡，从而可以接入两个交换机：一个是外网交换机，负责将客户端的请求路由到对应的LoginApp或BaseApp上；另一个是内网交换机，负责将LoginApp或BaseApp上的请求路由到其他角色的进程上。

bigworld中使用一个Space类来表示表示一个完整地图，然后Space又根据负载情况动态分割成一个或多个矩形区域Cell。每个Cell负责Space的矩形区域是不会重叠的，且所有Cell的矩形区域的并集就是这个Space的完整区域。同时一个Cell的区域并不是固定的，会根据其相邻区域之间的负载分布情况来动态调整。全局唯一的CellAppMgr管理所有的CellApp，也就是场景服务器进程，一组服务器一般会有数十个CellApp。然后每个CellApp都会有不定个数的Cell在进程内执行。所以Space与Cell的相关信息都会经由CellApp上传到CellAppMgr上，通过这些信息就可以维护Cell的创建、销毁以及边界调整。

当一个客户端想连接到游戏服务器时，涉及到的流程以及进程角色如下:

• 首先会连接到LoginApp上。LoginApp会验证客户端的登录信息，
• 如果验证成功且通过DBMgr从数据库里加载出来了对应的玩家数据，然后往BaseAppMgr发送这个玩家数据来请求创建对应实体。
• BaseAppMgr接收到这个请求之后就会筛选出合适的BaseApp，并通知选定的BaseApp来创建一个proxy对象，代表对应的玩家。
• BaseApp创建proxy对象完成之后，就会反向通知BaseAppMgr操作完成，然后BaseAppMgr又会通知回LoginApp。
• 此时LoginApp就可以通知原来的客户端当前登录操作已经成功，且会附带上对应的proxy对象的地址。

客户端接收到登录成功的返回之后，就会以这个proxy对象地址去连接对应的BaseApp，请求将这个proxy对象绑定到当前的客户端连接，绑定的同时会下发这个玩家所有的客户端可见属性，让客户端创建一个对应的对象。

但是此时还没有进入场景，如果玩家想进入场景与其他玩家进行互动，流程链依然很长：

• 首先客户端发送一个进入场景的请求到BaseApp上的对应proxy对象。
• proxy对象就会以此为基础构造一个进入场景的请求，里面填充场景编号、位置数据以及当前proxy对象属性里的Cell可见属性，并将这个请求发送到CellAppMgr。
• CellAppMgr接收到这个请求之后就会根据传入的场景编号获取对应的Space， 然后再根据出生位置来计算出覆盖这个区域的Cell，并将这个进入场景的请求发送到这个Cell所在的CellApp。
• CellApp找到对应的Cell后，就会以请求里携带的玩家的属性数据来创建一个真正的RealEntity，并绑定proxy所在的通信地址为数据下行通道。创建成功之后会将这个RealEntity的唯一标识符返回给CellAppMgr
• CellAppMgr再附带这个Cell的地址返回给proxy对象。
• 当proxy对象接收到对应的RealEntity的地址之后，就会绑定对应RealEntity的通信地址为数据上行通道。

当RealEntity需要给对应的客户端发消息的时候，会先通过与对应proxy之间的通道来传递消息， proxy接收到消息之后，解析发现这是一个发往客户端的消息，就会转发到绑定的客户端连接。同时如果客户端的玩家对象想往对应的RealEntity发送消息，则也需要通过proxy进行中转。

由于Cell的区域是动态调整的，所以Cell之间的边界是会发生变化的，同时玩家又是可以任意移动的，所以玩家的RealEntity并不是一直绑定在同一个Cell上。当一个RealEntity绑定到一个新的Cell时，就会通知新Cell来创建一个新的RealEntity，同时老的RealEntity退化为一个GhostEntity，并通知对应的proxy对象切换上行通道的地址为新的RealEntity。一个RealEntity上发生的所有客户端可见属性的变化都会广播到其管理的GhostEntity集合，同时一个RealEntity下发的其他Entity的状态数据不仅包括周围的RealEntity，还包括周围的GhostEntity。如何通过RealEntity/GhostEntity来将分布式的场景营造成一个支持无缝迁移的统一逻辑场景复杂度其实挺高的，这也是Bigworld的核心技术所在。

Unreal Engine 服务端介绍

Unreal Engine（后文中将简称UE）是由Epic Games开发的商用游戏引擎，最早于1998年发布，之后更新了多个版本。在2014年发布了虚幻四(后文简称UE4)，并在Github上公开了其源代码，这种开放使用的策略大大的加速了UE的推广。然后在2022年进一步的发布了UE5，其黑客帝国Demo令人印象深刻，使得UE5次世代大世界这个词组成为了一个不可分割的整体。使用UE开发的游戏有很多，近年来比较知名的有PUBG、堡垒之夜、黑神话悟空、三角洲行动等。而且UE不仅仅能做游戏，还在虚拟现实、增强现实、电影特效、建筑可视化等各种实时互动内容制作里大显身手。

UE引擎有一个跟其他引擎非常不一样的地方，就是他的服务端与客户端源代码是混合在一起的，打包的时候使用同一份代码进行打包。引擎里自动就维护好了客户端与服务端之间的通信协议和状态同步机制，不需要去额外的做开发工作，这种设定下打包出来的服务器叫做专属服务器Dedicated Server，简称DS。如果需要在运行时区分客户端与服务端，那么就需要在代码里进行判断。以官方UE4样例项目ShooterGame的一个函数AShooterCharacter::PostInitializeComponents为例，运行时可以通过GetLocalRole()来判断当前进程是否有这个AShooterCharacter的控制权，然后GetNetMode()这个接口来判断当前进程的角色是客户端还是服务端：

void AShooterCharacter::PostInitializeComponents()
{
	Super::PostInitializeComponents();

	if (GetLocalRole() == ROLE_Authority)
	{
		Health = GetMaxHealth();

		// Needs to happen after character is added to repgraph
		GetWorldTimerManager().SetTimerForNextTick(this, &AShooterCharacter::SpawnDefaultInventory);
	}

	// 省略一些代码

	// play respawn effects
	if (GetNetMode() != NM_DedicatedServer)
	{
		// 省略一些代码
	}
}

更为厉害的是在编辑器内可以通过不同的命令行参数来分别启动客户端与服务端，这样可以非常方便的在编辑器里进行开发与测试。这样前后端代码一体化开发的方式，再加上UE引擎自带的蓝图（Blueprint）这个图形化编程工具，极大提升了非程序人员对UE引擎的接受度。

不过这样的一体化开发方式也有一些弊端，特别是服务端，因为服务端也需要处理物理、移动、骨骼动画等性能消耗非常高的任务，导致服务端的性能受到很大的限制，单进程的玩家承载量也受到了很大的限制，基本只能在100左右。正是由于这种承载量的限制，UE引擎目前适用的联网游戏类型基本都是一些匹配对战的开房间游戏，在需要大量玩家同场景的MMORPG(Massively Multiplayer Online Role-Playing Game)类型游戏中则比较少见。

UE打包生成的DS单个进程基本只能服务一个地图场景，在不深度修改的情况下无法做到多地图共存。而且单个DS进程长时间运行下会有比较严重的内存占用，所以一般来说单个DS进程只能存活比较短的时间，粒度基本都是以小时为单位，无法像MMORPG那样单个进程可以持续运行数周直到版本维护。同时UE开发框架里基本只能写地图内的玩家交互逻辑，无法作为聊天、好友、商城的常见游戏服务的进程宿主，导致还需要使用cpp/python/go等语言额外开发一些联网服务，并修改UE引擎来接入与这些服务相通信的RPC。所以一般使用UE作为房间服务器的服务端架构基本会设计成这个样子:

在这个架构图里，前置的网关分为了两种：一种是对接大厅服务的网关，采用TCP协议；一种是对接DS进程的网关，采用UDP协议。大厅服务主要负责玩家的注册登录、匹配、好友、组队、聊天等功能，而DS进程主要负责游戏场景的运行与玩家的交互逻辑。玩家客户端在启动之后，先连接到大厅网关服务，执行登录验证工作，登录成功之后就可以在大厅内执行一些场景无关操作。如果玩家需要进入特定的场景来进行相关玩法，则需要执行一个比较长的交互链条：

• 首先需要得到大厅服务这边的许可。当大厅服务认为可以进入特定场景的时候，会通知DS管理进程DS Manager来获取一个可用的DS进程来加载这个场景。
• 其实这里还有一个类似于Bigworld的BWMachined的DS Agent进程，部署在所有能承载DS的物理机器上，用来管理本机的所有DS的拉起、销毁和负载上报，这些DS Agent进程都受到接受DS Manager的管理。大厅请求DS Manager分配一个新的DS，DS Manager接收到请求之后会去找一个机器负载比较小的DS Agent，通知其创建一个新的DS进程，并加载指定的地图，开启特定的玩法。
• 当DS Agent执行完这个DS进程的拉起操作之后，就会汇报给DS Manager，DS Manager就带上这个DS进程的地址信息通知回大厅。
• 大厅接收到目标DS的地址信息之后，先通知到DS网关，给当前玩家加上目标DS的绑定，然后再通知当前玩家的客户端可以去发起到DS网关的连接。
• 客户端接收到DS的连接允许信息之后，就会以UE的连接协议去连接到DS网关。注意此时客户端与大厅之间的连接是没有断的，所以此时客户端与服务器之间会同时存在两条连接通路。
• DS网关此时会验证这个客户端是否有绑定的DS目标进程，如果有则开启一个UDP中转会话，用来桥接客户端与DS进程之间的UDP通信。

有的读者可能注意到UE里也有无缝迁移Seamless Travel的概念，但是这里的无缝迁移指的是一个DS进程以无缝的形式切换地图，此时客户端不需要断开与DS进程之间的连接，玩家控制的角色Character、控制器PlayerController和状态PlayerState能够自动的绑定到新的World。相对应的DS的非无缝迁移就是在切换地图之前DS通知所有客户端断开连接，然后再加载新的地图，此时客户端进程则重新执行连接与登录流程，这就会带来Character、PlayerController和PlayerState在服务端的销毁以及重新创建。

UE引擎是非常庞大的，包含了非常多的概念与模块，而且代码更新也极其频繁。在我有限的工作经历中，引擎重新编译涉及到的编译单元从UE4.23的不到2000个增长到现在UE5.5的将近6000个。不过跟服务端相关的Gameplay和网络相关功能UE4与UE5之间的差异并不大，所以本书所讲解的UE相关内容附带的代码都是基于UE4的。如果读者之前没有接触过UE，可以先去阅读一下Epic中国社区经理大钊在知乎写的系列文章Inside UE4，补充一下知识储备。当然读者如果完全对UE不感兴趣，可以在后文的阅读中直接跳过UE相关的章节。如果不是工作需要查证相关问题的话，我自己都有点看不下去这些源代码。

Mosaic Game 服务端介绍

mosaic_game这个项目创建于2020年，但是其实其使用的核心组件在2018年就可以开始陆陆续续的开发了。当时我负责了项目内的行为树与AI，但是当时的编辑器是PyQt实现的，代码非常扭曲，导出的数据文件居然是基于Pickle的，更新功能难度非常大。备受折磨的我准备使用CPP + QT5来重写，将数据格式改成json，顺带体验一下什么叫做GUI开发。完成了行为树编辑器的重新迭代之后，为了方便日常开发中的行为树问题调试，就给这个编辑器添加了调试器的功能。整个行为树开发套件开源在huangfeidian/behavior_tree上。刚好项目内有一套基于树形结构的角色数值属性计算系统，就顺带以这个图形化的编辑器框架为基础，构造了属性公式编辑器，以及属性公式求值运行时，开源在huangfeidian/formula_tree。

当时项目组使用的服务端引擎是没有源代码的，我们日常开发只能使用Python来写业务逻辑。写Python写烦了就开始琢磨这个引擎的底层实现，特别是玩家属性在Python脚本里修改之后是怎么自动同步到客户端的。当时Bigworld代码还未开源，同时UE4引擎还在我的认知之外，自己对CPP还很热情，因此开始从零开始用CPP来实现一个属性自动同步系统。这个属性自动同步系统还是比较复杂的，为此开发了好几个库：基于json的序列化与反序列化库huangfeidian/any_container;基于libclang的CPP反射库huangfeidian/meta;最终的属性修改自动同步库huangfeidian/property_sync。

在开发huangfeidian/meta这个CPP反射库的过程中，发现这个库提供的元数据收集加代码生成功能不仅可以为上面的属性同步系统服务，还可以来构造基于json的RPC系统，甚至还可以以此来构造基于Actor/Component的Entity系统。构造完这些系统之后发现项目组使用的游戏服务器的大半功能都被覆盖了，为什么不去整个大活，仿造一个完整的游戏服务器！刚好在网络通信上我也有一些积累，基于boost/asio实现过一个带流量加密的http代理服务器huangfeidian/http-proxy。于是就有了mosaic_game这个项目的诞生，开始了我慢慢填坑的过程。

这个mosaic_game是基于纯CPP实现的，但是CPP的包管理功能很弱，不像Python有很多开箱即用的库，因此在开发这个项目的过程中，我需要自己实现很多功能。为了方便游戏内各个功能的验证与调试，避免每次都启动整个游戏服务器通过再添加日志来定位问题，后续功能的添加基本都是以库的形式来组织的，除了上述介绍的功能之外，主要的功能库还有:

• 将excel表格中的数据转换为json的工具库huangfeidian/typed_matrix。这个库里还引用了我写的另外两个小组件：用来规定和验证excel数据格式的huangfeidian/typed_string和用来读取excel文件的huangfeidian/xlsx_reader。
• 聚合了各种AOI计算方法的库huangfeidian/aoi。
• 聊天系统huangfeidian/chat。
• 事件和状态机库huangfeidian/events。
• 匹配系统huangfeidian/match_maker。
• 游戏内的Entity系统huangfeidian/entity_component_factory。
• Redis桥接库huangfeidian/http_redis。
• MongoDB桥接库huangfeidian/http_mongo。
• 基于RecastNavigation的异步寻路库huangfeidian/AsyncDetourCrowd。
• 分布式场景管理库huangfeidian/distributed_scene。

通过集成上述功能，并不断的完善基础的游戏玩法，mosaic_game逐渐有了一个完整的游戏服务器雏形，下图就是最终的服务端架构图：

在这个架构图里，除了外部提供的Mongo进程和Redis进程，主要的进程角色有：

• 管理进程mgr_server，全局唯一，负责管理整体服务器集群的启动与退出，每个其他进程启动之后都需要向mgr_server注册。当一个进程完全可用之后，会通知mgr_server，然后mgr_server会将这个新进程的信息广播到现有的所有进程。同时mgr_server负责收集各个进程的负载信息，以方便执行一些进程角色的负载均衡
• Redis代理进程redis_server，可以有多个，负责代理其他进程与Redis服务器进行交互，
• MongoDB代理进程db_server，可以有多个，负责代理其他进程与MongoDB数据库进行交互
• 服务进程service_server，可以有多个，作为承载社交等场景无关逻辑的容器，一个service_server内可以有多个service
• 场景进程space_server，可以有多个，作为承载游戏场景逻辑的容器，一个space_server内可以有多个space
• 网关进程gate_server，可以有多个，负责中转客户端与space_server之间的通信
• 地图进程map_server，可以有多个，负责管理游戏场景中的地图数据，目前只承担了地图寻路功能，一个map_server可以为多个场景提供地图服务

由于当前的设计里还没有集成消息总线，所以目前的进程间互联图看上去乱糟糟的，等以后有空了去使用消息总线来简化一下连接拓扑：

• 所有进程都需要连接到mgr_server
• 每个space_server和每个service_server都需要连接到一个redis_server和一个db_server
• 每个space_server都需要连接到所有的service_server，这样space_server上才知道任意一个service在哪一个service_server上
• 每两个space_server之间都需要有一个连接
• 每个space_server都需要连接到一个map_server
• 每个gate_server都需要连接到所有的space_server

在当前的服务端架构之下，只有gate_server是外网可以直接访问的，此时客户端的登录和进入场景的流程是这样的：

• 客户端以某种方式获取当前可用的gate_server列表，随机选择一个gate_server发送会话建立请求
• gate_server接受了这个连接之后，先执行加密握手，验证客户端的身份，创建一个会话，并商定后续的对称加密密钥
• gate_server随机选择一个space_server来创建这个客户端对应的服务端账号对象account_entity，然后属性同步系统会通知客户端来同步创建这个client_account，之后gate_server就会当作一个透明的通信中转代理的作用，因此不再提及
• 客户端client_account对象再发起一个登录请求,包含账号密码
• space_server上的对应account_entity收到登录请求后，将登录请求转发到登录服务login_service上
• login_service收到登录请求，通过db_server查询账号密码是否正确,并将验证结果下发到account_entity,如果正确则记录当前账号已经在线
• space_server上的account_entity收到登录验证结果后，根据验证结果来通知客户端登录成功或失败，如果成功则开始往db_server请求这个账号的角色列表数据，并进行角色列表的下发
• 客户端的client_account收到角色列表数据后，通知服务端account_entity来选择其中一个来创建角色
• space_server上的account_entity收到角色选择请求后，通过db_server查询所选角色的完整数据，并以此数据来创建角色对象player_entity
• space_server上的player_entity创建完成之后，属性同步系统会将player_entity的属性同步到客户端，通知客户端创建角色对象client_player
• 客户端的client_player创建完成之后，可以发送RPC到space_server上的player_entity来申请进入特定场景
• space_server上的player_entity收到进入场景请求后，会转发这个请求到场景服务space_service上
• space_service检查是否有对应的场景space_entity，如果没有则通过负载均衡选择一个space_server来创建一个新的场景space_entity
• space_service找到合适的space_entity之后，将这个player_entity添加到space_entity的玩家列表中，同时通知player_entity开始切换到目标space_entity中，
• space_server上的player_entity开始执行场景切换，这个切换可能发生在同一个space_server上，也可能发生在不同的space_server上，如果是不同的space_server则还涉及到迁移流程，这里就不去展开了
• space_server上的player_entity进入到新的场景之后，属性同步系统会将新场景的属性同步到客户端，通知客户端切换到新的场景
• 客户端的client_player收到场景切换通知后，开始执行场景切换，并填充场景属性数据，通知space_server上的player_entity场景切换完成
• space_server上的player_entity收到客户端场景切换完成通知后，AOI系统就开始工作，开始向客户端同步周围的其他entity的状态

上述流程虽然冗长，但是好在都是面条代码，逻辑复杂度都非常低，实现起来都比较简单，大概2022年就完成了这些内容。但是随着无缝迁移这个目标的引入，场景管理和属性同步系统就需要大修，这样就缝缝补补了一年多才搞好。现在看上去无缝迁移的核心问题都被解决了，但是后面我已经没有多少精力去做各种测试样例，所以不保证当前的mosaic_game能够像想象中的样子来正确支持无缝迁移， BUG应该会有很多。好在应该没有人会在实际项目中使用mosaic_game这个玩具，这样我就先心安理得的休息一段时间，等有空了再去补这些单元测试。

Mosaic Game 的进程生命周期

服务器启动流程

在之前的游戏服务器架构中我们已经初步的介绍了游戏服务器中的一些常见的进程角色，在mosaic_game中也基本复用了相关概念，在服务器集群中的进程角色有如下六种：

1. 管理进程，作为全局的单点进程，第一个执行启动，用来提供进程注册和服务发现等功能，其代码在roles/server/mgr_server目录下
2. 场景进程，用来承载各种局内玩法，作为各种entity与space的容器，其代码在roles/server/space_server目录下
3. 网关进程，用来中转客户端与服务端之间的消息，其代码在roles/server/gate_server目录下
4. 服务进程，用来承载各种局外玩法，如聊天、好友、排行榜等，其代码在roles/server/service_server目录下
5. 数据库进程，用来转接游戏内对数据库的读写请求，其使用的后端数据库为mongodb，其代码在roles/server/db_server目录下
6. 缓存进程，用来承接游戏内对一些缓存数据的读写请求，其使用的后端缓存为redis，其代码在roles/server/redis_server目录下
7. 地图进程，用来处理一些场景的寻路、物理、AOI等资源查询，其代码在roles/server/map_server下

在了解这些进程角色之后，就可以来介绍一下这些进程的启动流程了。我提供了一个非常简陋的服务器启动脚本deploy/scripts/run_servers.py，在这个python文件中，可以指定除了管理进程之外的其他进程的数量，同时还可以指定一些数据文件位置和一些外围服务所需的配置文件位置。在启动流程中，mgr_server会作为第一个进程起来，这个角色的进程只有一个实例。接下来依次会启动缓存进程、数据库进程、服务进程、网关进程、场景进程、地图进程：

cur_server_cmd = "nohup ../bin/mgr_server -c {0}  -l {1} -n mgr_server > /dev/null 2>{1}/mgr_server.log &".format(options.config_path, options.log_path)
os.system(cur_server_cmd)
sleep(1)
for i in range(1):
	cur_server_cmd = "nohup ../bin/redis_server -c {0}  -l {1} -n redis_server_{2}   -f {3}  > /dev/null 2>{1}/redis_server_{2}.log &".format(options.config_path,  options.log_path, i, options.redis_config)
	os.system(cur_server_cmd)

for i in range(options.db_num):
	cur_server_cmd = "nohup ../bin/db_server -c {0}  -l {1} -n db_server_{2}  -m {3}  > /dev/null 2>{1}/db_server_{2}.log &".format(options.config_path,  options.log_path, i, options.mongo_config)
	os.system(cur_server_cmd)

for i in range(options.service_num):
	cur_server_cmd = "nohup ../bin/service_server -c {0}  -l {1} -n service_server_{2} -d {3}> /dev/null 2>{1}/service_server_{2}.log &".format(options.config_path,  options.log_path, i, options.data_dir)
	os.system(cur_server_cmd)

for i in range(options.gate_num):
	cur_server_cmd = "nohup ../bin/gate_server -c {0}  -l {1} -n gate_server_{2} > /dev/null 2>{1}/gate_server_{2}.log &".format(options.config_path,  options.log_path, i)
	os.system(cur_server_cmd)

for i in range(options.game_num):
	cur_server_cmd = "nohup ../bin/space_server -c {0}  -l {1} -n space_server_{2} -d {3}> /dev/null 2>{1}/space_server_{2}.log &".format(options.config_path,  options.log_path, i, options.data_dir)
	os.system(cur_server_cmd)

for i in range(options.map_server_num):
	cur_server_cmd = "nohup ../bin/map_server -c {0}  -l {1} -n map_server_{2} -d {3}> /dev/null 2>{1}/map_server_{2}.log &".format(options.config_path,  options.log_path, i, options.data_dir)
	os.system(cur_server_cmd)
sleep(1)

开头的第一个sleep(1)是为了保证在其他进程初始化时当前的mgr_server已经初始化好，后面的sleep(1)是为了在自动化测试的时候避免出现客户端启动了但是服务器还没有准备好的情况。

上述启动的各种server进程对应的逻辑类都继承自include/stub/json_stub.h里提供的json_stub，roles/basic_client里提供的客户端也继承自json_stub。同时这个json_stub又继承自basic_stub，这个basic_stub就是mosaic_game中所有进程角色的逻辑框架。在进程启动时，会通过相关配置文件来初始化这个逻辑框架运行所需的一些基础信息：

basic_stub::basic_stub(boost::asio::io_context& in_io_con, const stub_info& in_local_server, const stub_info& in_upstream_server, std::size_t in_timeout, utility::ts_t in_timer_check_gap_ms, std::uint32_t async_thread_num)
	: m_io_context(in_io_con)
	, m_local_server(in_local_server)
	, m_upstream_server(in_upstream_server)
	, m_logger(utility::get_logger(in_local_server.name))
	, m_connection_timeout(in_timeout)
	, m_timer_check_gap_ms(in_timer_check_gap_ms)
	, m_local_name_ptr(std::make_shared<const std::string>(in_local_server.name))
	, m_upstream_name_ptr(std::make_shared<const std::string>(in_upstream_server.name))
	, m_start_ts(std::chrono::duration_cast<std::chrono::seconds>(std::chrono::system_clock::now().time_since_epoch()).count())
	, m_anchor_name_prefix(in_local_server.name + utility::rpc_anchor::seperator)
	, m_async_thread_num(async_thread_num)
{
	m_asio_wrapper = std::make_unique<asio_wrapper>(m_io_context);
}

这里的in_local_server与in_upstream_server都是stub_info类型，用来明确一个进程的名字、类型与端口信息：

struct stub_info
{
	std::string ip;
	std::uint16_t port;
	std::string rsa_key;
	std::string name;
	std::string upstream;
	std::uint16_t http_port;
	std::string type;
};

in_local_server代表当前进程的配置信息， in_upstream_server代表当前进程的上游进程对应的信息。除了mgr_server之外的其他进程在初始化的时候都需要传入对应的mgr_server作为上游管理进程，所以这里的ip,port字段其实只有mgr_server对应的stub_info才需要设置，其他时候保留为空。一个集群中的所有stub_info里的name不得重复，因为这个name字段就是此进程的唯一标识符。如果一个stub_info有上游，则upstream字段就是对应上游进程stub_info里的name。

在basic_stub的start函数里会使用create_router来初始化网络相关资源，然后根据upstream_server里的配置调用connect_to_server去对应的上游进程执行注册。同时调用start_accept根据local_server里的port,http_port来分别开启对正常游戏连接端口以及http调试端口的监听：

void basic_stub::start()
{
	m_logger->info("server start");
	
	create_router();
	if (!m_upstream_server.name.empty())
	{
		add_named_server(m_upstream_server);
		connect_to_server(m_upstream_server.name);
	}
	start_accept();
	m_asio_wrapper->m_timer.expires_from_now(std::chrono::seconds(1));
	m_asio_wrapper->m_timer.async_wait([this](const asio_error_code& error)
		{
			if(error)
			{
				m_logger->error("m_asio_wrapper timer error {}", error.message());
				exit(1);
			}
			this->main_loop();
		});
	keep_alive_callback();
	m_async_task_channels = std::vector<mutex_channel<std::function<void()>>>(m_async_thread_num);
	m_async_threads.reserve(m_async_thread_num);
	for(std::uint32_t i = 0; i< m_async_thread_num; i++)
	{
		m_async_threads.push_back(std::make_unique<std::thread>([this, i](){
			this->async_worker_loop(i);
		}));
	}
}

第一次启动的时候先用一个1s间隔的计时器去延迟启动框架主循环，keep_alive_callback则负责定期向上游进程发送心跳包来执行进程保活，最后再按照配置去创造一些后台worker线程来作为线程池备用。注意主循环的代码是跑在asio相关线程上的，这里的worker线程并不会处理任何asio相关的任务，asio的线程是在main函数里创建的，不归basic_stub维护。下面就是gate_server在main函数里启动asio线程的相关代码，这里设置为threads为1代表只开启一个asio网络线程:

std::uint8_t const threads = 1;
std::size_t expire_time = 30000;
asio::io_context ioc{ threads };

server::gate_server cur_server = server::gate_server(ioc, local_stub_info, upstream_stub_info, expire_time, utility::ts_t(20));
cur_server.start();
std::vector<std::thread> v;
v.reserve(threads);
for (auto i = threads; i > 0; --i)
	v.emplace_back(
		[&ioc]
		{
			ioc.run();
		});
for(auto& one_thread: v)
{
	one_thread.join();
}

事实上除了上面介绍的这些手动创建的线程之外，还会有一个由spdlog自动创建的日志线程，由于这个线程完全由spdlog托管，所以这里就不去深究了。

不同的进程角色在启动时不仅要执行这个start函数，还有一些自己的额外逻辑去处理。所以在子类中，一般会提供一个自己的do_start函数来执行各自的初始化流程，并在这个流程中去调用basic_stub::start：

1. 在map_server的do_start最为简单，只是初始化了内部的地图设置

void map_server::do_start(const map_config& in_map_config)
{
	m_map_config = in_map_config;
	json_stub::start();
	
}

2. 在redis_server的do_start函数中，会构造线程池去连接后端的redis集群

void redis_server::do_start(const redis_config &redis_servers, std::uint8_t worker_num)
{
	
	for (std::uint8_t i = 0; i < worker_num; i++)
	{
		auto cur_task_logger = utility::get_logger("redis_worker_" + std::to_string(i) );
		workers.push_back(std::make_shared<redis_worker>(redis_servers, redis_task_channels, cur_task_logger));
	}
	for (auto &one_worker : workers)
	{
		work_threads.emplace_back([=]()
									{ one_worker->run(); });
	}
	json_stub::start();
}

3. 在db_server中的do_start函数中，也会创建线程池去连接后端的mongodb集群

void db_server::do_start(const mongo_config& mongo_servers, std::uint8_t worker_num)
{
	auto cur_task_logger = utility::get_logger("mongo_worker");
	for (std::uint8_t i = 0; i < worker_num; i++)
	{
		
		workers.push_back(std::make_shared<mongo_worker>(mongo_servers, mongo_task_channels, cur_task_logger));
	}
	for (auto& one_worker : workers)
	{
		work_threads.emplace_back([=]()
			{
				one_worker->run();
			});
	}
	json_stub::start();
}

4. 在space_server中的do_start函数，负责初始化entity系统、manager系统以及配置表系统

void space_server::do_start()
{

	entity::entity_manager::instance().init();
	
	json_stub::start();
	manager_base::init_managers(this);
	misc::stuff_utils::init();
	global_config_mgr::instance();
}

5. 在service_server中的do_start函数，负责初始化自己的服务管理器以及本地单例manager

void service_server::do_start()
{
	m_service_mgr = &service::service_manager::instance();
	m_service_mgr->init();
	manager_base::init_managers(this);
	json_stub::start();
}

6. 在mgr_server中的do_start函数，负责启动一个创建service的计时器

void mgr_server::do_start()
{
	json_stub::start();
	m_service_create_check_timer = add_timer_with_gap(std::chrono::milliseconds(1000), [this]() {
		check_to_create_service();
		});
}

7. 在gate_server中的do_start函数， 负责启动一个计时器去定期清理无效的会话

void gate_server::do_start()
{
	json_stub::start();
	m_session_remove_timer = add_timer_with_gap(std::chrono::seconds(1), [this]() 
	{
		this->check_remove_session();
	});
}

服务器主循环

在框架主循环中main_loop，有五种信息需要处理，分别是：

1. 普通网络消息network::msg_task，对应的是服务器进程之间以及客户端与网关进程之间的业务消息
2. http消息http_utils::request，对应的是此进程开启的http服务接收的消息，目前主要作为调试命令使用
3. 业务计时器超时消息，对应的是游戏业务逻辑内的各种计时器，而不是asio提供的计时器
4. 连接控制消息connection_ctrl_msg，对应的是各进程之间的tcp连接的建立与断开消息
5. 主线程回调消息std::function<void()>，对应的是一些非主线程执行的业务回调，例如http请求以及异步线程池处理的相关业务

void basic_stub::main_loop()
{
	m_logger->flush();
	auto cur_msg_handler = [this](std::shared_ptr<network::net_connection> con, const network::msg_task& one_msg)
	{
		return this->on_msg(con, one_msg);
	};
	auto cur_http_handler = [this](const http_utils::request& req, msg_seq_t req_seq)
	{
		return this->on_http_request(req, req_seq);
	};

	auto cur_conn_ctrl_msg_handler = [this](const network::connection_ctrl_msg& msg)
	{
		return this->on_conn_ctrl_msg(msg);
	};
	std::uint64_t loop_total_ms = 0;
	do
	{
		on_new_frame();
		poll_mainloop_tasks();
		auto poll_begin_ts = utility::timer_manager::now_ts();
		m_router->poll_msg(cur_msg_handler);
		http::http_request_mgr::poll_request(cur_http_handler);
		poll_timers(utility::timer_manager::now_ts());
		m_router->poll_ctrl_msg(cur_conn_ctrl_msg_handler);
		auto poll_end_ts = utility::timer_manager::now_ts();
		loop_total_ms += poll_end_ts - poll_begin_ts;
		if (poll_end_ts - poll_begin_ts > m_high_load_threshold * m_timer_check_gap_ms)
		{
			continue;
		}
		else
		{
			break;
			
		}

	}while (!m_stopped);

	if(m_stopped)
	{
		//  暂时忽略进程退出相关代码
	}
	m_asio_wrapper->m_timer.expires_from_now(std::chrono::milliseconds(std::max(int64_t(1), int64_t(m_timer_check_gap_ms) - int64_t(loop_total_ms))));
	m_asio_wrapper->m_timer.async_wait([this](const asio_error_code& error)
		{
			(void)error;
			this->main_loop();
		});
	return;

}

在单次主循环处理完成之后，会检查此次主循环的执行时间。如果执行时间小于指定的阈值m_high_load_threshold * m_timer_check_gap_ms，则说明当前要处理的任务比较多，负载比较高，此时会继续执行一次逻辑主循环，直到单次逻辑主循环消耗的时间小于指定值或者外部发起了关服请求。在本批次的逻辑主循环执行完成之后，继续创建一个计时器去延迟执行下一次主循环，这里的延迟设置是为了尽量的保证主循环开始的间隔保持为预设的m_timer_check_gap_ms，同时避免主循环空跑造成的性能浪费。

由于网络相关逻辑由network_router管理，与basic_stub的业务逻辑相解耦，所以在处理这些网络消息的时候，外部需要提供相关的回调函数，也就是上面代码中构造的三个msg_handler。这里的on_msg负责处理进程之间的业务消息，on_ctrl_msg负责处理进程间的连接建立与断开，这两个函数的细节将在后续的网络细节相关章节中介绍。on_http_request则负责处理本地http服务器接收到的http请求，这里的http请求主要是一些运维指令相关，内部会调用on_gm_cmd去分发这些运维指令：

void json_stub::on_http_request(const http_utils::request& req, msg_seq_t req_seq)
{
	std::string cmd;
	json param;
	try
	{
		auto cur_json = json::parse(req.body);
		cur_json.at("cmd").get_to(cmd);
		cur_json.at("param").get_to(param);
	}
	catch(const std::exception& e)
	{
		auto reply = fmt::format("fail to parse http req for req uri {} content {} req_seq {} error {}", req.uri, req.body, req_seq, e.what());
		m_logger->info(reply);
		on_http_reply(req_seq, reply);
		return;
	}
	on_gm_cmd(cmd, param, req_seq);
	
}

服务器间连接

目前的流程中，所有的进程角色在启动后都会向其上游角色发起连接。上游连接建立成功之后，会在对应的on_connect函数里执行一些自定义的逻辑，其中最重要的就是通过set_stub_info这个指令告诉mgr_server当前连接发起方的进程角色是什么：

void json_stub::on_connect(std::shared_ptr<network::net_connection> connection)
{
	basic_stub::on_connect(connection);

	const auto& cur_connection_dest = get_connection_name(connection.get());
	auto cur_server_iter = m_named_servers.find(*cur_connection_dest);
	if (cur_server_iter == m_named_servers.end())
	{
		close_connection(connection);
		return;
	}


	json request_msg_1;
	request_msg_1["cmd"] = "set_stub_info";
	json local_info;
	local_info["stub_info"] = m_local_server;
	request_msg_1["param"] = local_info;
	std::shared_ptr<std::string> msg_ptr = std::make_shared<std::string>(request_msg_1.dump());
	m_router->push_msg(connection.get(), m_local_name_ptr, cur_connection_dest, msg_ptr, enums::packet_cmd_helper::encode(enums::packet_cmd::server_control, 0));
}

一个进程角色向mgr_server汇报set_stub_info后并不代表这个进程角色已经可用，只有其向mgr_server发送report_server_ready之后这个进程才能作为完整的角色来提供服务。gate_server的功能比较单一，因此在其on_connect函数中会直接向mgr_server汇报当前进程已经可用

void gate_server::on_connect(std::shared_ptr<network::net_connection> connection)
{
	json_stub::on_connect(connection);
	auto connection_name_ptr = connection->get_connection_name();
	auto cur_server_iter = m_named_servers.find(*connection_name_ptr);
	if (cur_server_iter == m_named_servers.end())
	{
		return;
	}
	if (cur_server_iter->second.type == "space_server")
	{
		m_connected_gameservers[*connection_name_ptr] = connection;
	}
	else if(cur_server_iter->second.type == "mgr_server")
	{
		json report_ready_info, temp_param;
		report_ready_info["cmd"] = "report_server_ready";
		temp_param["server_name"] = *m_local_name_ptr;
		temp_param["server_type"] = "gate_server";
		report_ready_info["param"] = temp_param;
		m_router->push_msg(connection.get(), m_local_name_ptr, {}, std::make_shared<std::string>(report_ready_info.dump()), enums::packet_cmd_helper::encode(enums::packet_cmd::server_control, 0));
	}
	
}

与gate_server类似的进程角色还有map_server,db_server,redis_server，都是上游连接建立之后立即汇报server_ready。但是space_server和service_server就复杂了一些，他们不能在连接到mgr_server之后立即汇报就绪状态，因为这些进程角色需要与其他进程角色相配合才能提供完整的角色服务。这些角色的进程在连接建立之后会向mgr_server请求资源进程角色列表：

void service_server::request_allocate_resource_server(const std::string& resource_server_type)
{
	json request_msg, request_param;
	request_msg["cmd"] = "request_allocate_resource";
	request_param["from_server_name"] = m_local_server.name;
	request_param["from_server_type"] = m_local_server.type;
	request_param["resource_server_type"] = resource_server_type;
	request_msg["param"] = request_param;
	auto msg_ptr = std::make_shared<std::string>(request_msg.dump());
	auto remote_name_ptr = std::make_shared<std::string>(m_upstream_server.name);
	if (!m_router->push_msg(m_local_name_ptr, remote_name_ptr, msg_ptr, enums::packet_cmd_helper::encode(enums::packet_cmd::server_control, 0)))
	{
		add_timer_with_gap(std::chrono::milliseconds(2 * 1000), [resource_server_type, this]() {
			request_allocate_resource_server(resource_server_type);
			});
	}
}
void service_server::on_connect(std::shared_ptr<network::net_connection> connection)
{
	json_stub::on_connect(connection);
	auto connection_name_ptr = get_connection_name(connection.get());
	if (*connection_name_ptr == m_upstream_server.name)
	{
		request_allocate_resource_server("db_server");
		request_allocate_resource_server("redis_server");
	}
}

在mgr_server中接收到这些资源申请请求之后，会选取当前资源池中连接数最少的进程来返回：

std::string mgr_server::allocate_resource_svr(std::unordered_map<std::string, resource_stub_info>& resource_svrs, const std::string& from_server_name)
{
	if(resource_svrs.empty())
	{
		return {};
	}
	std::vector<std::pair<std::string, std::size_t>> resource_server_loads;
	resource_server_loads.reserve(resource_svrs.size());
	for (const auto& one_pair : resource_svrs)
	{
		if(!one_pair.second.ready)
		{
			continue;
		}
		resource_server_loads.emplace_back(one_pair.first, one_pair.second.connected_svrs.size());
	}
	std::sort(resource_server_loads.begin(), resource_server_loads.end(), [](const std::pair<std::string, std::size_t>& a, const std::pair<std::string, std::size_t>& b)
		{
			return a.second < b.second;
		});
	auto dest_resource_svr = resource_server_loads[0].first;
	resource_svrs[dest_resource_svr].connected_svrs.insert(from_server_name);
	return dest_resource_svr;
}

当space_server与service_server收到分配的资源进程列表之后，会主动的向这些资源进程发起连接：

void space_server::on_reply_allocate_resource(std::shared_ptr<network::net_connection> con, std::shared_ptr<const std::string> dest, const json& msg)
{
	stub_info cur_resource_svr;
	std::string resource_server_type;
	std::string cur_err;
	try
	{
		msg.at("resource_server_type").get_to(resource_server_type);
		msg.at("errcode").get_to(cur_err);
		if(!cur_err.empty())
		{
			m_logger->warn("on_reply_allocate_resource errcode {}", cur_err);
			add_timer_with_gap(std::chrono::milliseconds(2 * 1000), [resource_server_type, this]() {
				request_allocate_resource_server(resource_server_type);
			});
			return;
		}
		msg.at("resource_svr").get_to(cur_resource_svr);
	}
	catch (std::exception& e)
	{
		m_logger->warn("on_reply_allocate_resource msg invalid {} error {}", msg.dump(4), e.what());
		return;
	}

	m_named_servers[cur_resource_svr.name] = cur_resource_svr;
	connect_to_server(cur_resource_svr.name);

}

当各项进程角色连接数量得到满足之后，就会向mgr_server汇报自己这个进程已经可用了。

void space_server::on_connect(std::shared_ptr<network::net_connection> connection)
{
	json_stub::on_connect(connection);
	auto connection_name_ptr = get_connection_name(connection.get());
	if (*connection_name_ptr == m_upstream_server.name)
	{
		request_allocate_counter("online_session");
		request_allocate_resource_server("db_server");
		request_allocate_resource_server("redis_server");
	}
	else
	{
		auto cur_server_iter = m_named_servers.find(*connection_name_ptr);
		if (cur_server_iter == m_named_servers.end())
		{
			return;
		}
		if (cur_server_iter->second.type == "db_server")
		{
			m_router->link_anchor_to_connection("db_server", connection.get());
			m_logger->info("m_connected_resource_servers add {}", *connection_name_ptr);
		}
		else if(cur_server_iter->second.type == "redis_server")
		{
			m_router->link_anchor_to_connection("redis_server", connection.get());
			m_logger->info("m_connected_resource_servers add {}", *connection_name_ptr);
		}
		else
		{
			return;
		}
		if(m_router->has_anchor("redis_server") && m_router->has_anchor("db_server"))
		{
			json report_ready_info, temp_param;
			report_ready_info["cmd"] = "report_server_ready";
			temp_param["server_name"] = *m_local_name_ptr;
			temp_param["server_type"] = m_local_server.type;
			report_ready_info["param"] = temp_param;
			m_router->push_msg( m_local_name_ptr,std::make_shared<std::string>(m_upstream_server.name), std::make_shared<std::string>(report_ready_info.dump()), enums::packet_cmd_helper::encode(enums::packet_cmd::server_control, 0));
		}
	}
}

当space_server变成ready之后，会向gate_server进行广播：

// 	void mgr_server::on_report_server_ready(std::shared_ptr<network::net_connection> con, std::shared_ptr<const std::string> dest, const json& msg)
json broadcast_msg, param;
std::vector<stub_info> temp_stub_infos;
temp_stub_infos.push_back(m_named_servers[server_name]);
broadcast_msg["cmd"] = "notify_server_ready";
param["servers"] = temp_stub_infos;
broadcast_msg["param"] = param;
auto cur_info = std::make_shared<std::string>(broadcast_msg.dump(4));
broadcast_to_space_svrs(cur_info);
if(server_type == "space_server")
{
	broadcast_to_gate_svrs(cur_info);
}

gate_server收到space_server的ready消息之后，会主动发起一个连接：

void gate_server::on_notify_server_ready(std::shared_ptr<network::net_connection> con, std::shared_ptr<const std::string> dest, const json& msg)
{
	std::vector<stub_info> ready_servers;
	try
	{
		msg.at("servers").get_to(ready_servers);
	}
	catch (std::exception& e)
	{
		m_logger->error("on_notify_server_ready fail to parse {} error {}", msg.dump(4), e.what());
		return;
	}
	for (const auto& one_server : ready_servers)
	{
		if (one_server.type != "space_server")
		{
			continue;
		}
		m_named_servers[one_server.name] = one_server;

		if (!m_router->has_connection_with_name(one_server.name))
		{
			connect_to_server(one_server.name);
		}

	}
}

当一个service_server变成ready之后，mgr_server并不会立即往这些service_server分配service去创建，而是等待一个计时器超时再去创建：

void mgr_server::check_to_create_service()
{
	if(m_stopped)
	{
		return;
	}
	if (m_service_create_check_timer.valid())
	{
		m_timer_mgr.cancel_timer(m_service_create_check_timer);
		m_service_create_check_timer.reset();

	}
	std::uint32_t need_server_num = 0;
	for (const auto& one_pair : m_services_to_create)
	{
		need_server_num = std::max(need_server_num, one_pair.second);
	}
	std::vector<std::string> ready_service_servers;
	for (const auto& one_pair : m_service_stub_infos)
	{
		if (one_pair.second.ready)
		{
			ready_service_servers.push_back(one_pair.first);
		}
	}
	if (need_server_num > ready_service_servers.size())
	{
		// m_logger->error("need_server_num {} is larger than m_service_stub_infos size {}", need_server_num, ready_service_servers.size());
		add_timer_with_gap(std::chrono::milliseconds(1000), [this]() {
			check_to_create_service();
			});
		return;
	}
	if (m_min_service_server_num > ready_service_servers.size())
	{
		// m_logger->error("m_min_service_server_num {} is larger than m_service_stub_infos size {}", m_min_service_server_num, ready_service_servers.size());
		add_timer_with_gap(std::chrono::milliseconds(1000), [this]() {
			check_to_create_service();
			});
		return;
	}

	std::unordered_map<std::string, std::vector<std::string>> temp_services_on_server;
	std::uint32_t temp_counter = 0;
	for (const auto& one_service_cluster : m_services_to_create)
	{
		for (std::uint32_t i = 0; i < one_service_cluster.second; i++)
		{
			auto cur_select_server = ready_service_servers[temp_counter % ready_service_servers.size()];
			temp_services_on_server[cur_select_server].push_back(one_service_cluster.first);
			request_create_service_on_server(cur_select_server, one_service_cluster.first, {});
			temp_counter++;
		}
	}
}

这样做的目的是为了等待所有的service_server都启动好之后再去平摊各种局外服务的创建，否则就可能出现所有服务都创建在同一个service_server的情况。

服务器关闭流程

服务器关闭指令需要运维人员手动去操作，通过mgr_server的http端口进行通知，在deploy/scripts/stop_servers.py中我们通过curl指令来执行关服通知：

with open(options.config_path, "r") as config_file:
	config_detail = json.load(config_file)
	mgr_server_ip = config_detail["mgr_server"]["ip"]
	mgr_server_http_port = config_detail["mgr_server"]["http_port"]
	curl_cmd = "curl -H \"Accept: application/json\" -H \"Content-type: application/json\" -X GET -d '{\"cmd\": \"stop\", \"param\": {}}'  "
	curl_cmd += "http://" + mgr_server_ip + ":" + str(mgr_server_http_port) + "/gm_cmd/GET/"
	os.system(curl_cmd)

mgr_server接收到这个运维指令之后，会调用notify_stop来开始执行关服逻辑：

void json_stub::on_gm_cmd(const std::string& cmd, const json& param, msg_seq_t req_seq)
{
	if(cmd == "stop")
	{
		notify_stop();
		json reply_json;
		reply_json["params"] = param;
		auto reply_str = reply_json.dump() + "\r\n";

		http::http_request_mgr::finish_request(req_seq, reply_str);
	}
}

void basic_stub::notify_stop()
{
	if(m_stopped)
	{
		m_logger->error("notify stop while already stopping");
		return;
	}
	m_logger->warn("notify_stop");
	m_stopped = true;
	m_stop_report_ts = m_stop_begin_ts = std::chrono::steady_clock::now();
	stop_begin();
}

void basic_stub::stop_begin()
{
	m_logger->info("stop_begin");
}

在notify_stop中我们会设置m_stopped这个bool变量为true，代表当前进程正在关服流程中，至于具体的关服细节则依赖于各个进程角色的重载实现。

在进程主循环中，如果发现自己是关服过程中，则开始定期检查关服过程是否已经执行完毕，如果执行完毕则开始关闭所有的监听端口以及计时器，等待io_service的自然退出：

// void basic_stub::main_loop()
if(m_stopped)
{
	auto stop_check_ts = std::chrono::steady_clock::now();
	bool with_stop_log = false;
	std::chrono::duration<double> stop_elapsed_time = stop_check_ts - m_stop_report_ts;
	if(stop_elapsed_time.count() > m_stop_alert_duration)
	{
		m_logger->warn("stop check fail after {} seconds", stop_elapsed_time.count());
		with_stop_log = true;
		m_stop_report_ts = stop_check_ts;
	}
	if(check_stop_finish(with_stop_log))
	{
		on_stop_finish();
		return;
	}
}

void basic_stub::on_stop_finish()
{
	m_router->disconnect_all();
	m_asio_wrapper->m_acceptor.close();
	m_asio_wrapper->m_timer.cancel();
	m_http_server->stop();
	m_logger->warn("on_stop_finish");
}

这里的check_stop_finish主要检查其他进程发起的连接以及http连接是否为0，都是0的时候再等待所有的线程池退出：

bool basic_stub::check_stop_finish(bool with_log)
{
	
	std::uint32_t has_upstream_server = 0;
	if(!m_upstream_server.name.empty())
	{
		has_upstream_server = 1;
	}
	if(m_router->get_active_connection_count() > has_upstream_server)
	{
		if(with_log)
		{
			m_logger->debug("router->get_active_connection_count() fail remain {}", m_router->get_active_connection_count());
		}
		return false;
	}
	if(m_http_server)
	{
		if(m_http_server->get_session_count() != 0)
		{
			if(with_log)
			{
				m_logger->debug("_http_server->get_session_count() fail");
			}
			return false;
		}
	}
	if(m_finished_async_thread_counter != m_async_thread_num)
	{
		if(with_log)
		{
			m_logger->debug("m_finished_async_thread_counter fail");
		}
		return false;
	}
	for(auto& one_thread_ptr : m_async_threads)
	{
		one_thread_ptr->join();
	}
	m_async_threads.clear();
	return true;
}

从前面小章节可知各种角色的服务器的就绪状态其实是有依赖的，这种逻辑依赖不仅影响服务器的启动，还影响服务器的关闭，强制关闭所有的连接可能会导致服务器状态没有正确的保存到数据库。所以mgr_server关服的时候是分阶段来通知各个进程角色去退出的的，阶段变量存储在m_stop_stage中，首先通知的是gate_server：

void mgr_server::stop_begin()
{
	json_stub::stop_begin();

	json sync_msg;
	sync_msg["cmd"] = "notify_stop";
	sync_msg["param"] = json::object_t();
	auto cur_stop_msg = std::make_shared<std::string>(sync_msg.dump());
	// 先通知所有的gate 退出
	broadcast_to_gate_svrs(cur_stop_msg);
	
	m_stop_stage = stop_stage::wait_gate_server_destroy;
	return;
}

gate_server在接收到这个的通知请求之后，会通知所有的客户端服务器关闭，让客户端主动的断开所有的连接，同时拒绝掉后续的所有新客户端发出来的连接：

void gate_server::stop_begin()
{
	json_stub::stop_begin();
	std::vector<std::string> temp_entities;
	for(const auto& one_pair: m_eid_to_conn_id)
	{
		temp_entities.push_back(one_pair.first);
	}
	json::object_t notify_msg;
	notify_msg["msg"] = "server_close";
	for(const auto& one_dest: temp_entities)
	{
		request_client_close_impl(one_dest, notify_msg);
	}
	add_timer_with_gap(std::chrono::seconds(1), [this]()
	{
		m_router->disconnect_all();
	});
}

这里会同时开启一个延迟计时器去强制关闭所有的连接，包括客户端、space_server、mgr_server。mgr_server会收到gate_server的断线通知，当所有的gate_server都退出之后，开始进入第二阶段，通知所有玩家都下线：

void mgr_server::on_gate_all_destroyed()
{
	m_logger->warn("on_gate_all_destroyed");
	if(m_stop_stage != stop_stage::wait_gate_server_destroy)
	{
		return;
	}
	m_stop_stage = stop_stage::wait_account_logout;
	json sync_msg;
	sync_msg["cmd"] = "request_logout_all_accounts";
	sync_msg["param"] = json::object_t();
	auto cur_stop_msg = std::make_shared<std::string>(sync_msg.dump());
	// 通知 所有的space server account准备logout
	broadcast_to_space_svrs(cur_stop_msg);

	// 等待login_service汇报所有玩家都已经下线
}

当这个request_logout_all_accounts消息发送到space_server之后，所有的在线账号都会通知其对应的在线玩家去执行存库后下线操作，如果没有对应在线玩家则直接执行账号下线操作：

void space_server::on_request_logout_all_accounts(std::shared_ptr<network::net_connection> con, std::shared_ptr<const std::string> from, const json& msg)
{
	// 通知所有的account 准备logout
	auto cur_accounts = entity::entity_manager::instance().get_entities_by_exact_type<entity::account_entity>();
	utility::rpc_msg logout_msg;
	logout_msg.cmd = "request_logout_account";
	for(auto one_account: cur_accounts)
	{
		one_account->on_rpc_msg(logout_msg);
	}
	utility::rpc_msg request_msg;
	request_msg.cmd = "request_check_accounts_empty";
	call_service("login_service", request_msg);
}

void account_entity::request_logout_account(const utility::rpc_msg& msg)
{
	if(m_statem.active_state_name() == "logout_account")
	{
		return;
	}
	if(m_player_id.empty())
	{
		utility::rpc_msg request_msg;
		request_msg.cmd = "request_logout_account";
		call_service("login_service", request_msg);
		dispatcher().dispatch(enums::event_category::account, "logout");
		m_statem.change_to("logout_account");
	}
	else
	{
		utility::rpc_msg request_msg;
		request_msg.cmd = "request_logout_player";
		call_player(request_msg);
		m_statem.change_to("logout_account");
	}
	
}

当所有在线账号都下线之后，login_service会通知mgr_server下线阶段执行完成：

void service_server::report_accounts_all_logout()
{
	m_logger->info("report_accounts_all_logout");
	json report_info;
	report_info["cmd"] = "report_accounts_all_logout";

	report_info["param"] = json::object_t();
	m_router->push_msg(m_local_name_ptr, std::make_shared<std::string>(m_upstream_server.name), std::make_shared<std::string>(report_info.dump()), enums::packet_cmd_helper::encode(enums::packet_cmd::server_control, 0));
}

在此之后mgr_server通知所有的service执行存库后退出操作，：

void mgr_server::on_accounts_all_logout(std::shared_ptr<network::net_connection> con, std::shared_ptr<const std::string> from, const json& msg)
{
	m_logger->warn("on_accounts_all_logout");
	if(m_stop_stage != stop_stage::wait_account_logout)
	{
		return;
	}
	m_stop_stage = stop_stage::wait_service_destroy;
	// login_service汇报所有账号都已经下线
	json sync_msg;
	sync_msg["cmd"] = "notify_stop";
	sync_msg["param"] = json::object_t();
	auto cur_stop_msg = std::make_shared<std::string>(sync_msg.dump());
	// 通知所有的service准备退出
	broadcast_to_service_svrs(cur_stop_msg);
	broadcast_to_space_svrs(cur_stop_msg);
}

同时由于所有的账号都下线了，所有的space_server也不再被需要，开始执行退出流程：

void space_server::stop_begin()
{
	manager_base::stop_managers();
	json_stub::stop_begin();
}

service_server接收到stop通知后，会遍历当前进程上的所有service，执行存库后退出操作：

void service_server::stop_begin()
{
	m_logger->warn("service server {} stop_begin ", *m_local_name_ptr);
	auto cur_services = m_service_mgr->get_all_servicies();
	for(auto one_pair: cur_services)
	{
		one_pair.second->notify_stop();
	}
	manager_base::stop_managers();
}

当一个service完成了自己的退出逻辑之后，会通知到mgr_server:

void service_server::destroy_service(service::base_service* cur_service)
{
	json report_destroy_info, temp_param;
	report_destroy_info["cmd"] = "report_service_destroyed";
	temp_param["service_type"] = cur_service->m_base_desc.m_type_name;
	temp_param["service_id"] = cur_service->global_id();
	report_destroy_info["param"] = temp_param;
	m_router->push_msg( m_local_name_ptr, std::make_shared<std::string>(m_upstream_server.name), std::make_shared<std::string>(report_destroy_info.dump()), enums::packet_cmd_helper::encode(enums::packet_cmd::server_control, 0));
	m_services_by_id.erase(cur_service->global_id());
	
	m_services_to_destroy.push_back(cur_service);
}

当mgr_server记录的所有service都退出后，开始通知space_server与service_server执行连接清理并退出:

void mgr_server::on_service_all_destroyed()
{
	m_logger->warn("on_service_all_destroyed");
	if(m_stop_stage != stop_stage::wait_service_destroy)
	{
		return;
	}
	m_stop_stage = stop_stage::wait_space_server_destroy;
	// service 都退出之后 通知 space server 与 service server主动退出
	json sync_msg;
	sync_msg["cmd"] = "notify_clear_connection";
	sync_msg["param"] = json::object_t();
	auto cur_stop_msg = std::make_shared<std::string>(sync_msg.dump());

	broadcast_to_space_svrs(cur_stop_msg);
	broadcast_to_service_svrs(cur_stop_msg);
	// 等待 所有的 space 服务器都退出
}

当所有的space_server都下线之后，最后通知所有的资源服务器执行下线操作：

void mgr_server::on_space_server_all_destroyed()
{
	m_logger->warn("on_space_server_all_destroyed");
	if(m_stop_stage != stop_stage::wait_space_server_destroy)
	{
		return;
	}
	m_stop_stage = stop_stage::wait_connection_clear;
	json sync_msg;
	sync_msg["cmd"] = "notify_stop";
	sync_msg["param"] = json::object_t();
	auto cur_stop_msg = std::make_shared<std::string>(sync_msg.dump());
	// 通知所有的资源服务器准备退出
	broadcast_to_map_svrs(cur_stop_msg);
	broadcast_to_redis_svrs(cur_stop_msg);
	broadcast_to_db_svrs(cur_stop_msg);
}

由于redis与db才是最后负责数据落地的角色，所以这两种进程的退出条件优先判断是否所有的读写任务都已经完成：

bool db_server::check_stop_finish(bool with_log)
{
	
	if (!mongo_task_channels.tasks_all_finished())
	{
		if(with_log)
		{
			m_logger->debug("mongo_task_channels check fail");
		}
		return false;
	}
	else
	{
		for (auto& one_worker : workers)
		{
			one_worker->notify_stop();
		}
		return json_stub::check_stop_finish(with_log);
	}
}

bool redis_server::check_stop_finish(bool with_log)
{

	if (!redis_task_channels.tasks_all_finished())
	{
		if (with_log)
		{
			m_logger->debug("redis_task_channels check fail");
		}
		return false;
	}
	else
	{
		for (auto &one_worker : workers)
		{
			one_worker->notify_stop();
		}
		return json_stub::check_stop_finish(with_log);
	}
}

当所有的连接都断开之后，mgr_server的check_stop_finish就会返回true，并最终导致mgr_server进程退出。

BigWorld 的进程生命周期

启动流程

在Bigworld里，每种进程都有一个专门的main.cpp文件来定义其启动时的main函数,其文件内容大同小异,只是bwMainT< >的模板参数不同。其模板参数就是进程的App类，例如CellApp、BaseApp、LoginApp等:

// server\dbappmgr\main.cpp
int BIGWORLD_MAIN( int argc, char * argv[] )
{
	return bwMainT< DBAppMgr >( argc, argv );
}
// server\loginapp\main.cpp
int BIGWORLD_MAIN( int argc, char * argv[] )
{
	return bwMainT< LoginApp >( argc, argv );
}

这里的BIGWORLD_MAIN宏定义了进程的main函数,来封装一些通用的处理流程,例如初始化Bigworld的资源和配置,解析命令行参数等，最后调用bwMainT< >来启动进程的主循环:

#define BIGWORLD_MAIN 										\
bwMain( int argc, char * argv[] );							\
int main( int argc, char * argv[] )							\
{															\
	BW_SYSTEMSTAGE_MAIN();									\
	BWResource bwresource;									\
	BWResource::init( argc, (const char **)argv );			\
	BWConfig::init( argc, argv );							\
	bwParseCommandLine( argc, argv );						\
	return bwMain( argc, argv );							\
}															\
int bwMain

这个bwMainT< >模板函数负责进程运行环境的准备，包括初始化全局事件驱动器EventDispatcher、网络资源interface以及注册进程退出的信号处理函数signalProcessor。初始化好了这三个重要组件之后，继续利用模板函数doBWMainT< >来启动进程的主循环:

// lib\server\bwservice.hpp
template <class SERVER_APP>
int bwMainT( int argc, char * argv[], bool shouldLog = true )
{
	Mercury::EventDispatcher dispatcher;

	// Find the internal interface IP by querying BWMachined
	if (!Mercury::MachineDaemon::queryForInternalInterface(
			ServerApp::discoveredInternalIP ))
	{
		WARNING_MSG( "bwMainT: "
			"Unable to determine internal interface via BWMachineD query.\n" );
	}

	BW::string internalInterfaceName =
			getBWInternalInterfaceSetting( SERVER_APP::configPath() );

	Mercury::NetworkInterface interface( &dispatcher,
			Mercury::NETWORK_INTERFACE_INTERNAL, 0,
			internalInterfaceName.c_str() );

	SignalProcessor signalProcessor( dispatcher );

	BW_MESSAGE_FORWARDER3( SERVER_APP::appName(), SERVER_APP::configPath(), 
		/*ENABLED=*/shouldLog, dispatcher, interface );

	START_MSG( SERVER_APP::appName() );

	if (internalInterfaceName != Mercury::NetworkInterface::USE_BWMACHINED)
	{
		CONFIG_WARNING_MSG( "internalInterface set to '%s' in bw.xml. "
				"This option is deprecated. It is recommended to not set this "
				"value. The default behaviour is to use the same interface as "
				"bwmachined. This is controlled by the broadcast route.\n",
			internalInterfaceName.c_str() );
	}

	int result = doBWMainT< SERVER_APP >( dispatcher, interface, argc, argv );

	INFO_MSG( "%s has shut down.\n", SERVER_APP::appName() );

	return result;
}

这个doBWMainT的实现就非常简单了，主要就是创建一个SERVER_APP的实例，然后调用其runApp方法来启动进程的主循环:

template <class SERVER_APP>
int doBWMainT( Mercury::EventDispatcher & dispatcher,
		Mercury::NetworkInterface & interface,
		int argc, char * argv[] )
{
	if (!ServerAppConfig::init( SERVER_APP::Config::postInit ))
	{
		CONFIG_ERROR_MSG( "Failed to initialise configuration options. "
				"See earlier error message for details.\n" );
		return -1;
	}

	SERVER_APP serverApp( dispatcher, interface );
	serverApp.setBuildDate( __TIME__, __DATE__ );

	return serverApp.runApp( argc, argv ) ? EXIT_SUCCESS : EXIT_FAILURE;
}

每种进程都有一个对应的App类，这些App类都继承自ServerApp类。ServerApp类是Bigworld进程管理的核心类，ServerApp::runApp来调用init初始化相关组件:

/**
 *	This method runs this application.
 */
bool ServerApp::runApp( int argc, char * argv[] )
{
	// calculate the clock speed
	stampsPerSecond();

	bool result = false;

	if (this->init( argc, argv ))
	{
		INFO_MSG( "---- %s is running ----\n", this->getAppName() );
		result = this->run();
	}
	else
	{
		ERROR_MSG( "Failed to initialise %s\n", this->getAppName() );
	}

	this->fini();

	interface_.prepareForShutdown();

#if ENABLE_PROFILER
	g_profiler.fini();
#endif

	return result;
}

这里的init方法负责初始化性能监控g_profiler，退出信号处理函数pSignalHandler_，以及FD的最大打开数，并设置网络接口的日志等级:

/**
 *	Initialisation function.
 *	
 *	This needs to be called from subclasses' overrides.
 */
bool ServerApp::init( int argc, char * argv[] )
{
	PROFILER_SCOPED( ServerApp_init );

	bool runFromMachined = false;

	for (int i = 1; i < argc; ++i)
	{
		if (strcmp( argv[i], "-machined" ))
		{
			runFromMachined = true;
		}
	}

	INFO_MSG( "ServerApp::init: Run from bwmachined = %s\n",
			watcherValueToString( runFromMachined ).c_str() );

#if ENABLE_PROFILER
	if (ServerAppConfig::hasHitchDetection())
	{
		g_profiler.setProfileMode( Profiler::SORT_BY_TIME, false );
	}
#endif

	pSignalHandler_.reset( this->createSignalHandler() );

	// Handle signals
	this->enableSignalHandler( SIGINT );

	this->raiseFileDescriptorLimit( ServerAppConfig::maxOpenFileDescriptors() );

	interface_.verbosityLevel( ServerAppConfig::isProduction() ?
		Mercury::NetworkInterface::VERBOSITY_LEVEL_NORMAL :
		Mercury::NetworkInterface::VERBOSITY_LEVEL_DEBUG );

	return true;
}

在完成了init之后，ServerApp::runApp接下来会调用run方法来启动进程的主循环EventDispatcher::processOnce：

/**
 *	This is the default implementation of run. Derived classes to override
 *	this to implement their own functionality.
 */
bool ServerApp::run()
{
	mainDispatcher_.processUntilBreak();

	this->onRunComplete();

	return true;
}

/**
 *	This method call processContinuously until breakProcessing is called.
 */
void EventDispatcher::processUntilBreak()
{
	this->processContinuously();
	pErrorReporter_->reportPendingExceptions( true /* reportBelowThreshold */ );
}

/**
 * 	This method processes events continuously until interrupted by a call to
 * 	breakProcessing.
 *
 *	@see breakProcessing
 */
void EventDispatcher::processContinuously()
{
	breakProcessing_ = false;

	while (!breakProcessing_)
	{
		this->processOnce( /* shouldIdle */ true );
	}
}

在processOnce方法中，会先处理频繁任务processFrequentTasks，然后处理定时器processTimers，随后是性能统计processStats，最后处理网络事件processNetwork：

/**
 *	This method processes the current events.
 *
 *	@param shouldIdle	If set to true, this method will block until the next
 *						timer is due if there is nothing waiting on the network.
 *
 *	@return 			The number of network events processed.
 */
int EventDispatcher::processOnce( bool shouldIdle /* = false */ )
{
	breakProcessing_ = false;

	this->processFrequentTasks();

	if (!breakProcessing_)
	{
		this->processTimers();
	}

	this->processStats();

	if (!breakProcessing_)
	{
		return this->processNetwork( shouldIdle );
	}

	return 0;
}

这个processFrequentTasks方法会通过FrequentTasks::process方法来处理所有通过mainDispatcher_.addFrequentTask注册过来的FrequentTask，目前主要用到这个FrequentTask的地方不多，只有mysql,signal和http相关的任务会注册到FrequentTasks中:

/**
 *	This method calls doTask on all registered FrequentTasks objects.
 */
void EventDispatcher::processFrequentTasks()
{
	PROFILER_SCOPED( processFrequentTasks );
	pFrequentTasks_->process();
}

// lib\db_storage_mysql\mysql_database.cpp
/**
 *	This method implements the FrequentTask method.
 */
void MySqlDatabase::doTask()
{
	bgTaskManager_.tick();
}
// lib\server\signal_processor.cpp
void SignalProcessor::doTask()
{ 
	this->dispatch(); 
}
/**
 *	Handle frequent task trigger.
 */
void SignalProcessor::dispatch()
{
	const Signal::Set allSignals( Signal::Set::FULL );
	Signal::Blocker blocker( allSignals );

	int sigNum = SIGMIN;
	while (sigNum <= SIGMAX)
	{
		if (signals_.isSet( sigNum ))
		{
			this->dispatchSignal( sigNum );
		}

		++sigNum;
	}

	signals_.clearAll();
}

在processTimers方法里会调用TimeQueue::process方法来处理全局计时器队列pTimeQueue_里所有到期的定时器任务，这个方法会返回处理的定时器任务数量，我们可以通过这个数量来判断是否有定时器任务到期：

/**
 *	This method processes outstanding timers.
 */
void EventDispatcher::processTimers()
{
	PROFILER_SCOPED( processTimers );
	numTimerCalls_ += pTimeQueue_->process( timestamp() );
}

在processNetwork方法中，会调用Poller::processPendingEvents方法来处理所有等待在网络上的事件，这个方法会返回处理的事件数量，我们可以通过这个数量来判断是否有网络事件发生：

/**
 *	This method processes any activity on the network.
 *
 *	@param shouldIdle If set to true, this method will block until the next
 *	timer is due if there is nothing waiting on the network.
 *	@return	Number of file descriptors that triggered handlers.
 */
int EventDispatcher::processNetwork( bool shouldIdle )
{
	PROFILER_SCOPED( processNetwork );
	// select for network activity until earliest timer
	double maxWait = shouldIdle ? this->calculateWait() : 0.0;

	return pPoller_->processPendingEvents( maxWait );
}

目前这个Poller的实现有三种, 就是常见的Linux系统上的select,poll和epoll：

1. SelectPoller 调用select来等待文件描述符的相关状态改变，并执行对应的事件处理函数。
2. PollPoller 调用poll来等待文件描述符的相关状态改变，并执行对应的事件处理函数。
3. EPoller 调用epoll_wait来等待文件描述符的相关状态改变，并执行对应的事件处理函数。

运行的时候根据编译选项会选择不同的Poller实现，默认是SelectPoller，如果编译选项中定义了HAS_POLL，则会选择PollPoller，如果定义了HAS_EPOLL，则会选择EPoller。

/**
 *	This static method creates an appropriate EventPoller. It may use select or
 *	epoll.
 */
EventPoller * EventPoller::create()
{
#if defined( HAS_POLL )
	return new PollPoller();

#elif defined( HAS_EPOLL )
	return new EPoller();

#else // !defined( HAS_EPOLL )
	return new SelectPoller();

#endif // defined( HAS_EPOLL )
}

这里的事件循环里并没有处理具体的业务逻辑，只是处理了网络事件和定时器任务。ServerApp的具体子类负责利用这里提供的Task和Timer组件来插入自身的业务逻辑。以最简单的DBApp为例，在其连接到数据库之后会调用initTimers，这里会开启一个固定间隔且重复调用的计时器来驱动GameTick,在其超时处理函数handleTimeout里用来定时调用advanceTime方法来推进时间：

/**
 *  This method sets up required timers.
 *
 *  @return true on success, false otherwise.
 */
bool DBApp::initTimers()
{
    MF_ASSERT( status_.status() < DBStatus::RUNNING );

    // A one second timer to check all sorts of things, including whether to
    // start the server running if we are waiting for other components to
    // be ready.
    statusCheckTimer_ = mainDispatcher_.addTimer( 1000000, this,
            reinterpret_cast< void * >( TIMEOUT_STATUS_CHECK ),
            "StatusCheck" );

    // NOTE: DBApp's time is not synchronised with the rest of the cluster.
    gameTimer_ = mainDispatcher_.addTimer( 1000000/Config::updateHertz(), this,
            reinterpret_cast< void * >( TIMEOUT_GAME_TICK ),
            "GameTick" );

    return true;
}

/**
 *  This method handles timer events. It is called every second.
 */
void DBApp::handleTimeout( TimerHandle handle, void * arg )
{
    switch (reinterpret_cast< uintptr >( arg ))
    {
    case TIMEOUT_GAME_TICK:
        this->advanceTime();
        break;

    case TIMEOUT_STATUS_CHECK:
        this->checkStatus();
        break;
    }
}

这个advanceTime由ServerApp这个基类来提供，主要内容就是推进游戏时间，提供游戏的Tick机制,也就是常说的游戏帧的概念。内部会调用onTickPeriod、onEndOfTick、onStartOfTick、onTickProcessingComplete等回调函数，以及调用callUpdatables方法来更新所有Updatable对象。

/**
 *	This method increments the game time.
 */
void ServerApp::advanceTime()
{
	if (lastAdvanceTime_ != 0)
	{
		double tickPeriod = stampsToSeconds( timestamp() - lastAdvanceTime_ );

		this->onTickPeriod( tickPeriod );
	}

	lastAdvanceTime_ = timestamp();

	this->onEndOfTick();

	++time_;

#if ENABLE_PROFILER
	g_profiler.tick();
#endif

	this->onStartOfTick();

	this->callUpdatables();

	this->onTickProcessingComplete();
}

这里的几个Tick相关函数目前基本都是空实现，主要是为了预留扩展点，方便子类来实现自己的逻辑。只有ServerApp::onTickPeriod提供了一个基础实现，就是对过长的帧时间进行警告并记录:

virtual void onTickPeriod( double tickPeriod );

/*
	* This method gives subclasses a chance to act at the end of a tick
	* immediately before the current server time is incremented.
	*/
virtual void onEndOfTick() {};

/*
	* This method gives subclasses a chance to act at the beginning of a tick
	* immediately after the current server time is incremented.
	*/
virtual void onStartOfTick() {};

/*
	* This method gives subclassses a chance to act at the end of ServerApp's
	* tick processing, immediately before control returns to the caller of
	* ServerApp::advanceTime();
	*/
virtual void onTickProcessingComplete() {};

/*
 * This method gives subclasses the tick period each time advanceTime is
 * called
 */
void ServerApp::onTickPeriod( double tickPeriod )
{
	if (tickPeriod * ServerAppConfig::updateHertz() > 2.0)
	{
		WARNING_MSG( "ServerApp::onTickPeriod: "
					"Last game tick took %.2f seconds. Expected %.2f.\n",
				tickPeriod, 1.0/ServerAppConfig::updateHertz() );
	}

#if ENABLE_PROFILER
	if (ServerAppConfig::hasHitchDetection() &&
		((tickPeriod * ServerAppConfig::updateHertz()) >
			ServerAppConfig::hitchDetectionThreshold()))
	{
		WARNING_MSG( "Service::onTickPeriod: "
			"Server hitch detected, creating JSON dump.\n" );
		g_profiler.dumpThisFrame();
	}
#endif
}

注册互联

Bigworld服务器集群里会存在各种角色的进程，包括DBApp、GameApp、LoginApp等。对于DBApp、BaseApp、CellApp等角色的进程，在一个集群里会有一个到多个具体的实例。为了统一管理这些不定数量的角色实例，Bigworld里设计了DBAppMgr、BaseAppMgr和CellAppMgr等管理器。在这些DBApp、BaseApp、CellApp等角色的进程启动之后，会将这些进程注册到对应的Mgr实例里。这个注册机制依赖于ServerApp上提供的一个ManagerAppGateway类型的对象，默认情况下ServerApp::pManagerAppGateway返回空，只有在这些需要被统一管理的进程角色里会对这个接口进行重载:

// lib\server\server_app.hpp
virtual ManagerAppGateway * pManagerAppGateway() { return NULL; }
// server\baseapp\baseapp.hpp
ManagerAppGateway * pManagerAppGateway() /* override */
{
	return &baseAppMgr_;
}
// server\cellapp\cellapp.hpp
ManagerAppGateway * pManagerAppGateway() /* override */
{
	return &cellAppMgr_;
}
// server\dbapp\dbapp.hpp
ManagerAppGateway * pManagerAppGateway() /* override */
{
	return &dbAppMgr_;
}

这个ManagerAppGateway负责当前进程与对应的管理进程之间的注册与反注册，构造的时候需要提供通信用的网络通道networkInterface和反注册用的RPC retireAppIE：

ManagerAppGateway::ManagerAppGateway( Mercury::NetworkInterface & networkInterface,
			const Mercury::InterfaceElement & retireAppIE ) :
		channel_( networkInterface, Mercury::Address::NONE ),
		retireAppIE_( retireAppIE )
{
	MF_ASSERT( retireAppIE_.lengthStyle() == Mercury::FIXED_LENGTH_MESSAGE );
	MF_ASSERT( retireAppIE_.lengthParam() == 0 );
}

// server\dbapp\dbappmgr_gateway.cpp
DBAppMgrGateway::DBAppMgrGateway( Mercury::NetworkInterface & interface ) :
	ManagerAppGateway( interface, DBAppMgrInterface::retireApp )
{}

// server\cellapp\cellappmgr_gateway.cpp
CellAppMgrGateway::CellAppMgrGateway( Mercury::NetworkInterface & interface ) :
	ManagerAppGateway( interface, CellAppMgrInterface::retireApp )
{}

// server\baseapp\baseappmgr_gateway.cpp
BaseAppMgrGateway::BaseAppMgrGateway( Mercury::NetworkInterface & interface ) :
	ManagerAppGateway( interface, BaseAppMgrInterface::retireApp )
{}

这个ManagerAppGateway构造函数并不直接负责注册，依赖于外部来调用ManagerAppGateway::init来初始化底层的通道Channel。接口需要提供能与目标ManagerApp通信的interfaceName，通过这个interfaceName来查询对应的通信地址：

bool ManagerAppGateway::init( const char * interfaceName, int numRetries, float maxMgrRegisterStagger )
{
	if (!almostZero( maxMgrRegisterStagger ))
	{
		const float MICROSECONDS_IN_SECOND = 1000000.0f;

		// Spread starting time of processes within a tick to avoid possible network peaks during startup
		BWRandom rand( mf_getpid() );
		uint32 delay = static_cast<uint>( rand( 0.0f, maxMgrRegisterStagger ) * MICROSECONDS_IN_SECOND );

		if (delay > 0)
		{
			DEBUG_MSG( "ManagerAppGateway::init: "
					"Manager Registration Stagger mode is active: maxMgrRegisterStagger: %f s. "
					"Delaying process start for %f s.\n",
				maxMgrRegisterStagger, delay / MICROSECONDS_IN_SECOND );

			usleep( delay );
		}
	}

	Mercury::Address addr;

	Mercury::Reason reason =
		Mercury::MachineDaemon::findInterface( interfaceName,
			0, addr, numRetries );

	if (reason == Mercury::REASON_SUCCESS)
	{
		channel_.addr( addr );
	}

	// This channel is irregular until we start the game tick timer.
	this->isRegular( false );

	return reason == Mercury::REASON_SUCCESS;
}

这个Mercury::MachineDaemon::findInterface函数负责查询interfaceName对应的通信地址addr，具体做法是向当前进程所属的MachineDaemon发送一个广播消息ProcessStatsMessage，查询所有能提供interfaceName的进程的通信地址，获取第一个返回的通信地址:

/**
 * 	This method finds a specified interface on the network.
 * 	WARNING: This function always blocks.
 *
 *	@param name			Only interfaces with this name are considered.
 *	@param id			Only interfaces with this ID are considered, if
 *						negative all are considered.
 *	@param addr			Output address of the found interface.
 *	@param retries		The number of retries if no interface is found.
 *	@param verboseRetry	Flag for versbose output on retry.
 *	@param pHandler		Handler to process ProcessStatsMessages responses. If
 *						NULL, the default FindFirstInterfaceHandler is used.
 *
 * 	@return	A Mercury::REASON_SUCCESS if an interface was found, a
 * 	Mercury::REASON_TIMER_EXPIRED if an interface was not found, other
 * 	Mercury::Reasons are returned if there is an error.
 */
Reason findInterface( const char * name, int id,
		Address & addr, int retries, bool verboseRetry,
		IFindInterfaceHandler * pHandler )
{
	ProcessStatsMessage pm;
	pm.param_ = pm.PARAM_USE_CATEGORY |
		pm.PARAM_USE_UID |
		pm.PARAM_USE_NAME |
		(id <= 0 ? 0 : pm.PARAM_USE_ID);
	pm.category_ = pm.SERVER_COMPONENT;
	pm.uid_ = getUserId();
	pm.name_ = name;
	pm.id_ = id;

	IFindInterfaceHandler * pDefaultHandler = NULL;

	if (!pHandler)
	{
		pDefaultHandler = new FindFirstInterfaceHandler();
		pHandler = pDefaultHandler;
	}

	int attempt = 0;

	retries = std::max( retries, 1 );

	while (++attempt <= retries)
	{
		Reason reason = pm.sendAndRecv( 0, BROADCAST, pHandler );

		if (reason != REASON_SUCCESS)
		{
			return reason;
		}

		if (pHandler->hasError())
		{
			return REASON_GENERAL_NETWORK;
		}

		Address result = pHandler->result();

		if (result != Address::NONE)
		{
			addr = result;
			return REASON_SUCCESS;
		}

		if (verboseRetry)
		{
			INFO_MSG( "MachineDaemon::findInterface: "
					"Failed to find %s for UID %d on attempt %d.\n",
				name, pm.uid_, attempt );
		}

		// Sleep a little because sendAndReceiveMGM() is too fast now! :)
#if defined( PLAYSTATION3 )
		sys_timer_sleep( 1 );
#elif !defined( _WIN32 )
		sleep( 1 );
#else
		Sleep( 1000 );
#endif
	}

	if (pDefaultHandler)
	{
		bw_safe_delete( pDefaultHandler );
	}

	return REASON_TIMER_EXPIRED;
}

这个findInterface的核心点在于ProcessStatsMessage的广播，需要探究一下这个消息是如何通知到当前服务器所在集群的MachineDaemon的。

Mercury::Reason MachineGuardMessage::sendAndRecv( uint32 srcip, uint32 destaddr,
	ReplyHandler *pHandler )
{
	BW_GUARD;
	// Set up socket
	Endpoint ep;
	ep.socket( SOCK_DGRAM );

	if (!ep.good() || ep.bind( 0, srcip ) != 0)
	{
		return Mercury::REASON_GENERAL_NETWORK;
	}

	return this->sendAndRecv( ep, destaddr, pHandler );
}

/**
 *	This method sends this MachineGuardMessage message to the bwmachined at the
 *	input address. The reply messages received are handled by the provided
 *	handler.
 *
 *	Note: If sending to BROADCAST, REASON_TIMER_EXPIRED will be returned if not
 *	all bwmachined daemons reply, even if some are successful.
 */
Mercury::Reason MachineGuardMessage::sendAndRecv( Endpoint &ep, uint32 destaddr,
	ReplyHandler *pHandler )
{
	BW_GUARD;

	if (destaddr == BROADCAST)
	{
		ep.setbroadcast( true );
	}

	char recvbuf[ MGMPacket::MAX_SIZE ];

	MachineGuardResponseChecker responseChecker;

	int countdown = 3;
	while (countdown--)
	{
		if (!this->sendto( ep, htons( PORT_MACHINED ), destaddr,
				MGMPacket::PACKET_STAGGER_REPLIES ))
		{
			ERROR_MSG( "MachineGuardMessage::sendAndRecv: "
					"Failed to send entire MGM (#%d tries left)\n",
				countdown );
			continue;
		}
		// 省略很多代码
	}
	ERROR_MSG( "MachineGuardMessage::sendAndRecv: timed out!\n" );
	return Mercury::REASON_TIMER_EXPIRED;
}

这里的sendto函数调用的时候，第二个参数是端口号PORT_MACHINED，这个端口号是bwmachined的监听端口，用于接收MachineGuardMessage消息。第三个参数是目标机器的地址，当前执行findInterface的时候被设置为了const uint32 BROADCAST = 0xFFFFFFFF;，表示广播发送。这个广播的UDP消息包只会发送到当前本地网络里的所有设备，不会被路由器转发到其他网段。这样粗暴的使用广播数据包来执行服务发现的方法真是令人叹为观止！这种方法的成功依赖于局域网内有开启PORT_MACHINED的监听的进程，目前Bigworld里使用了一个单独的BWMachineD进程来监听这个端口

/**
 * Discover the broadcast interface to use and init all the endpoints.
 */
void BWMachined::initNetworkInterfaces()
{
	// Determine which network interface will be sending broadcast messages
	if (broadcastAddr_ == 0 && !this->findBroadcastInterface())
	{
		syslog( LOG_CRIT, "Failed to determine default broadcast interface. "
					"Make sure UDP ports %d and %d are not firewalled and that "
					"your broadcast route is set correctly. e.g. "
					"/sbin/ip route add broadcast 255.255.255.255 dev eth0",
			PORT_MACHINED, PORT_BROADCAST_DISCOVERY );
		exit( EXIT_FAILURE );
	}

	if (!ep_.good() ||
		 ep_.bind( htons( PORT_MACHINED ), broadcastAddr_ ) == -1)
	{
		syslog( LOG_CRIT, "Failed to bind socket to '%s'. %s.",
							inet_ntoa((struct in_addr &)broadcastAddr_),
							strerror(errno) );
		exit( EXIT_FAILURE );
	}
	ep_.setbroadcast( true );

	if (!epLocal_.good() ||
		 epLocal_.bind( htons( PORT_MACHINED ), LOCALHOST ) == -1)
	{
		syslog( LOG_CRIT, "Failed to bind socket to (lo). %s.",
							strerror(errno) );
		exit( EXIT_FAILURE );
	}

	if (!epBroadcast_.good() ||
		 epBroadcast_.bind( htons( PORT_MACHINED ), BROADCAST ) == -1)
	{
		syslog( LOG_CRIT, "Failed to bind socket to '%s'. %s.",
							inet_ntoa((struct in_addr &)BROADCAST),
							strerror(errno) );
		exit( EXIT_FAILURE );
	}

	cluster_.ownAddr_ = broadcastAddr_;
}

这个BWMachineD进程在每个物理机上都有一个实例，然后多个实例之间通过PORT_MACHINED端口进行通信，从而组成一个集群cluster。

当一个ServerApp启动的时候，会将当前进程的地址、端口、pid、网络角色等信息打包成一个MachineGuardMessage消息，然后发送到本机的PORT_MACHINED端口。

// server\baseapp\baseapp.cpp
/**
 *  This method does the portion of the init after this app has been
 *  successfully added to the BaseAppMgr.
 */
bool BaseApp::finishInit( const BaseAppInitData & initData,	
		BinaryIStream & stream )
{
	// 省略一些代码
		if (isServiceApp_)
	{
		BaseAppIntInterface::registerWithMachinedAs( "ServiceAppInterface",
				this->intInterface(), id_ );
	}
	else
	{
		BaseAppIntInterface::registerWithMachined( this->intInterface(), id_ );
	}
}

// server\baseappmgr\baseappmgr.cpp
/**
 *	This method initialises this object.
 *
 *	@return True on success, false otherwise.
 */
bool BaseAppMgr::init( int argc, char * argv[] )
{
	// 省略一些代码

	Mercury::Reason reason = 
		BaseAppMgrInterface::registerWithMachined( interface_, 0 );
}

// server\cellapp\cellapp.cpp
/**
 *  This method handles the portion of init after registering with the
 *  CellAppMgr.
 */
bool CellApp::finishInit( const CellAppInitData & initData )
{
	// 省略一些代码
	CellAppInterface::registerWithMachined( this->interface(), id_ );
}

// server\cellappmgr\cellappmgr.cpp
/**
 *	The initialisation method.
 */
bool CellAppMgr::init( int argc, char * argv [] )
{
	// 省略一些代码

	{
		CellAppMgrInterface::registerWithInterface( interface_ );

		Mercury::Reason reason =
			CellAppMgrInterface::registerWithMachined( interface_, 0 );
	}
}

// server\dbapp\dbapp.cpp
/**
 *  This method performs initialisation for our newly received DBApp ID.
 *
 *  @return true on success, false otherwise.
 */
bool DBApp::initAppIDRegistration()
{
    // 省略一些代码

    if (DBAppInterface::registerWithMachined( interface_, id_ ) !=
            Mercury::REASON_SUCCESS)
    {
        NETWORK_ERROR_MSG( "DBApp::initAppIDRegistration: "
            "Unable to register with interface. Is machined running?\n" );
        return false;
    }

}

// server\dbappmgr\dbappmgr.cpp

bool DBAppMgr::init( int argc, char * argv [] )
{
	// 省略一些代码

	reason =
		DBAppMgrInterface::registerWithMachined( interface_, 0 );
}

// server\loginapp\loginapp.cpp
/**
 *	This method completes initialisation after registration to DBAppMgr.
 *
 *	@param appID 				The LoginApp ID.
 *	@param dbAppAlphaAddress 	The address of DBApp Alpha.
 */
bool LoginApp::finishInit( LoginAppID appID,
		const Mercury::Address & dbAppAlphaAddress )
{
	DEBUG_MSG( "LoginApp::finishInit: id %d (DBApp Alpha: %s)\n",
		appID, dbAppAlphaAddress.c_str() );

	id_ = appID;
	dbAppAlpha_.addr( dbAppAlphaAddress );

	Mercury::Reason reason =
		LoginIntInterface::registerWithMachined( this->intInterface(), id_ );

	// 省略一些代码
}

上述各种Interface::registerWithMachined执行的都是NetworkInterface::registerWithMachined：

/**
 *	This method is used to register or deregister an interface with the machine
 *	guard (a.k.a. machined).
 */
Reason NetworkInterface::registerWithMachined(
		const BW::string & name, int id )
{
	return this->interfaceTable().registerWithMachined( this->address(),
					name, id );
}
Reason InterfaceTable::registerWithMachined( const Address & addr )
{
	return MachineDaemon::registerWithMachined( addr,
			name_, id_ );
}

Reason InterfaceTable::registerWithMachined( const Address & addr,
				const BW::string & name, int id )
{
	name_ = name;
	id_ = id;
	return this->registerWithMachined( addr );
}

最后这里的MachineDaemon::registerWithMachined就是往LOCALHOST的BWMachineD发送注册信息的过程:

/**
 *	This function registers a socket with BWMachined.
 */
Reason registerWithMachined( const Address & srcAddr,
		const BW::string & name, int id, bool isRegister )
{
	if (name.empty())
	{
		return REASON_SUCCESS;
	}

#ifdef MF_SERVER
	// Do not call blocking reply handler after registering with bwmachined as
	// other processes can now find us and send other messages.
	BlockingReplyHandler::safeToCall( false );
#endif

	ProcessMessage pm;

	pm.param_ = (isRegister ? pm.REGISTER : pm.DEREGISTER) |
		pm.PARAM_IS_MSGTYPE;
	pm.category_ = ProcessMessage::SERVER_COMPONENT;
	pm.port_ = srcAddr.port;
	pm.name_ = name;
	pm.id_ = id;

	pm.majorVersion_ = BWVersion::majorNumber();
	pm.minorVersion_ = BWVersion::minorNumber();
	pm.patchVersion_ = BWVersion::patchNumber();

	ProcessMessageHandler pmh;

	// send and wait for the reply
	const uint32 destAddr = LOCALHOST;

	Reason response = pm.sendAndRecv( srcAddr.ip, destAddr, &pmh );

	return pmh.hasResponded_ ? response : REASON_TIMER_EXPIRED;
}

注意到这里只是往LOCALHOST发送注册消息，而不是广播发送到PORT_MACHINED端口，这样做是没什么问题的。因为目前BWMachineD之间并不会执行信息共享，所以只需要往LOCALHOST发送注册消息即可。只要查询的时候使用的是广播接口0xffffffff就没问题,因为广播消息总是可以发送到同一个集群里的各个物理机器。

当BWMachineD收到注册消息后，会将其记录到其内部的数组procs_中，这个数组里的每个元素都代表本机里能提供注册服务的一个进程:

/**
 *  @internal
 *  A ProcessStatsMessage is a more detailed version of ProcessMessage that
 *  provides CPU and memory usage info about the process.
 */
class ProcessStatsMessage : public ProcessMessage
{
public:
	uint8			cpu_;	//!< Process cpu as proportion of max
	uint8			mem_;	//!< Mem usage as proportion of max

	ProcessStatsMessage() : ProcessMessage(), cpu_( 0 ), mem_( 0 )
	{
		message_ = MachineGuardMessage::PROCESS_STATS_MESSAGE;
	}

	virtual ~ProcessStatsMessage() {}

	virtual const char *c_str() const;

protected:
	virtual void writeImpl( BinaryOStream &os );
	virtual void readImpl( BinaryIStream &is );
};
struct ProcessInfo
{
	ProcessInfo() { starttime = 0; }
	HighResStat cpu, mem;
	int affinity;

	ProcessStatsMessage m;

	// Time (since OS boot) that the process was started
	unsigned long int starttime;

	// Platform specific implementation
	void init( const ProcessMessage &pm );
};

BW::vector< ProcessInfo > procs_;

当接收到findInterface对应的查询请求PROCESS_STATS_MESSAGE的时候，只需要遍历procs_数组，找到所有符合条件的进程，然后将其信息打包成一个ProcessStatsMessage发送回查询请求者:

// bool BWMachined::handleMessage( Endpoint & ep, sockaddr_in & sin,
//	MachineGuardMessage & mgm, MGMPacket & replies )

case MachineGuardMessage::PROCESS_STATS_MESSAGE:
{
	ProcessStatsMessage &query = static_cast< ProcessStatsMessage& >( mgm );

	// Find the processes this matches
	bool found = false;
	for (BW::vector< ProcessInfo >::iterator it = procs_.begin();
			it != procs_.end(); ++it)
	{
		ProcessInfo &pi = *it;
		if (pi.m.matches( query ))
		{
			// Update load and mem stats on the MGM for this process
			SystemInfo &si = systemInfo_;
			uint64 cpuDiff = std::max(
				si.cpu[ pi.affinity % si.nCpus ].max.delta(),
				(uint64)1 );

			if (pi.affinity >= (int)si.nCpus)
				syslog( LOG_ERR, "ProcessInfo (%s) has invalid affinity %d",
					pi.m.c_str(), pi.affinity );

			ProcessStatsMessage &reply = pi.m;
			reply.cpu_ = (uint8)(pi.cpu.delta()*0xff / cpuDiff);
			reply.mem_ = (uint8)(pi.mem.cur()*0xff / si.mem.max.cur());

			// Add reply to the stream
			reply.copySeq( query );
			replies.append( reply );
			found = true;
		}
	}

	// If nothing found, send back a message with pid == 0 so that recv
	// loops can terminate on client side
	if (!found)
	{
		query.pid_ = 0;
		query.outgoing( true );
		replies.append( query );
	}

	return true;
}

当findInterface返回的时候, MgrAppGateway就可以根据结果来正确的获取目标MgrApp的地址信息，并根据地址信息来创建到MgrApp的channel，后续可以通过这个channel来发送注册消息。例如BaseAppMgrGateway上就暴露了一个add方法，来通知到对应的BaseAppMgr进程:

void BaseAppMgrGateway::add( const Mercury::Address & addrForCells,
		const Mercury::Address & addrForClients, bool isServiceApp,
		Mercury::ReplyMessageHandler * pHandler )
{
	Mercury::Bundle	& bundle = channel_.bundle();

	BaseAppMgrInterface::addArgs * args =
		(BaseAppMgrInterface::addArgs*)bundle.startStructRequest(
			BaseAppMgrInterface::add, pHandler );

	args->addrForCells = addrForCells;
	args->addrForClients = addrForClients;
	args->isServiceApp = isServiceApp;

	channel_.send();
}

这个add的调用发生在BaseApp::init函数的最末尾:

/**
 *	This method initialises this object.
 *
 *	@param argc	The number of elements in argv.
 *	@param argv	An array of argument strings.
 */
bool BaseApp::init( int argc, char * argv[] )
{
	if (!this->EntityApp::init( argc, argv ))
	{
		return false;
	}
	// 省略很多逻辑

	// Add ourselves to the BaseAppMgr.  Init will continue once the reply to
	// this message is received.  This object deletes itself.
	new AddToBaseAppMgrHelper( *this );

	return true;
}

/**
 *  This class helps to add this app to the BaseAppMgr.
 */
class AddToBaseAppMgrHelper : public AddToManagerHelper
{
public:
	AddToBaseAppMgrHelper( BaseApp & baseApp ) :
		AddToManagerHelper( baseApp.mainDispatcher() ),
		app_( baseApp )
	{
		// Auto-send on construction.
 		this->send();
	}


	void handleFatalTimeout()
	{
		ERROR_MSG( "AddToBaseAppMgrHelper::handleFatalTimeout: Unable to add "
				"%s to BaseAppMgr. Terminating.\n", app_.getAppName());
		app_.shutDown();
	}


	void doSend()
	{
		app_.baseAppMgr().add( app_.intInterface().address(), 
			app_.extInterface().address(), app_.isServiceApp(), this );
	}


	bool finishInit( BinaryIStream & data )
	{
		// Call send now so that any pending ACKs are sent now. This gives
		// finishInit a little longer before resends occur. Should not rely
		// on this as BaseAppMgr already thinks this BaseApp is running
		// normally.
		// TODO:BAR
		app_.baseAppMgr().send();

		BaseAppInitData initData;
		data >> initData;

		return app_.finishInit( initData, data );
	}

private:
	BaseApp & app_;
};

由于BaseApp,CellApp,DbApp都有注册到对应的MgrApp的需求，所以这里还抽象出来一个基类AddToManagerHelper，来统一处理注册逻辑。这个基类的send方法会调用到子类实现的doSend方法，来发送注册消息，对于BaseApp而言就是调用BaseAppMgrGateway::add方法。只有对应的MgrApp响应了这个Add注册请求之后，对应的App角色的finishInit接口才会执行，此时这个App相当于切换到了可用状态，可以承接后续的业务处理了。一般来说所有的游戏帧的启动都是在这个finishInit里，下面就是CellApp::finishInit的逻辑:

/**
 *  This method handles the portion of init after registering with the
 *  CellAppMgr.
 */
bool CellApp::finishInit( const CellAppInitData & initData )
{
	// Make sure that nothing else is read in the main thread.
	BWResource::watchAccessFromCallingThread( true );

	if (int32( initData.id ) == -1)
	{
		ERROR_MSG( "CellApp::finishInit: "
				"CellAppMgr refused to let us join.\n" );
		return false;
	}

	id_ = initData.id;
	this->setStartTime( initData.time );
	baseAppAddr_ = initData.baseAppAddr;
	dbAppAlpha_.addr( initData.dbAppAlphaAddr );
	isReadyToStart_ = initData.isReady;

	// Attach ourselves to an ID server (in this case, the Alpha DBApp).
	if (!idClient_.init( &this->dbAppAlpha(),
			DBAppInterface::getIDs,
			DBAppInterface::putIDs,
			IDConfig::criticallyLowSize(),
			IDConfig::lowSize(),
			IDConfig::desiredSize(),
			IDConfig::highSize() ))
	{
		ERROR_MSG( "CellApp::finishInit: Failed to get IDs\n" );
		return false;
	}

	timeoutPeriod_ = initData.timeoutPeriod;

	// Send app id to loggers
	LoggerMessageForwarder::pInstance()->registerAppID( id_ );

	if (isReadyToStart_)
	{
		this->startGameTime();
	}
	else
	{
		// Let startup() message handler start the game timer.
		isReadyToStart_ = true;
	}

	CONFIG_INFO_MSG( "\tCellApp ID            = %d\n", id_ );
	CONFIG_INFO_MSG( "\tstarting time         = %.1f\n",
		this->gameTimeInSeconds() );

	CellAppInterface::registerWithMachined( this->interface(), id_ );

	// We can safely register a birth listener now since we have mapped the
	// interfaces we are serving.
	Mercury::MachineDaemon::registerBirthListener(
			this->interface().address(),
			CellAppInterface::handleCellAppMgrBirth, "CellAppMgrInterface" );

	// init the watcher stuff
	char	abrv[32];
	bw_snprintf( abrv, sizeof(abrv), "cellapp%02d", id_ );
	BW_REGISTER_WATCHER( id_, abrv, "cellApp",
			mainDispatcher_, this->interface().address() );

	int pythonPort = BWConfig::get( "cellApp/pythonPort",
						PORT_PYTHON_CELLAPP + id_ );
	this->startPythonServer( pythonPort, id_ );

	INFO_MSG( "CellApp::finishInit: CellAppMgr acknowledged our existence.\n" );

	return true;
}

在上面的startGameTime会开启游戏帧定时器:

/**
 *	This method is called when we are ready to start the game timer.
 */
void CellApp::startGameTime()
{
	INFO_MSG( "CellApp is starting\n" );

	MF_ASSERT( !gameTimer_.isSet() && (pTimeKeeper_ == NULL) );
	MF_ASSERT( cellAppMgr_.isInitialised() );

	// start the game timer
	gameTimer_ = this->mainDispatcher().addTimer(
						1000000/Config::updateHertz(), this,
							reinterpret_cast< void * >( TIMEOUT_GAME_TICK ),
							"GameTick" );

	lastGameTickTime_ = timestamp();
	gettimeofday( &oldTimeval_, NULL );
	mainDispatcher_.clearSpareTime();
	this->calcTransientLoadTime();

	pTimeKeeper_ = new TimeKeeper( interface_, gameTimer_, time_,
		Config::updateHertz(), cellAppMgr_.addr(),
		&CellAppMgrInterface::gameTimeReading,
		id_, Config::maxTickStagger() );

	// Now we're sending load updates to the CellAppMgr regularly
	cellAppMgr_.isRegular( true );
}

当gameTimer_成功创建之后，当前CellApp的所有前期准备工作就都做完了，游戏帧定时器也开始运行了，在CellApp的handleTimeout方法中会通过handleGameTickTimeSlice来间接调用之前介绍的ServerApp::advanceTime来驱动各个Tick相关的回调:

/**
 *	This method handles timeout events.
 */
void CellApp::handleTimeout( TimerHandle /*handle*/, void * arg )
{
	switch (reinterpret_cast<uintptr>( arg ))
	{
		case TIMEOUT_GAME_TICK:
			this->handleGameTickTimeSlice();
			break;

		case TIMEOUT_TRIM_HISTORIES:
			this->handleTrimHistoriesTimeSlice();
			break;

		case TIMEOUT_LOADING_TICK:
		{
			// 省略一些代码
		}
	}
}

/**
 *	This method handles the game tick time slice.
 */
void CellApp::handleGameTickTimeSlice()
{
	// 省略一些代码

	this->updateBoundary();

	// Increment the time - we are now into the quantum of the next tick
	this->advanceTime();

	// 省略一些代码
}

对于LoginApp来说，虽然没有对应的LoginAppMgr，但是由于其业务重度依赖于数据库，所以在完成基础的初始化init之后，需要连接到DBApMgr来完成必要的初始化工作，:

bool LoginApp::init( int argc, char * argv[] ) /* override */
{
	// 省略一些代码
	// This calls back on finishInit().
	new AddToDBAppMgrHelper( *this );
	// 省略一些代码
}

在这个注册函数里会发送一个DBAppMgrInterface::addLoginApp的请求到dbAppMgr，在注册完成之后才会调用到LoginApp::finishInit来开启游戏帧Tick计时器:

class AddToDBAppMgrHelper : public AddToManagerHelper
{
public:

	/**
	 *	Constructor.
	 *
	 *	@param loginApp 	The LoginApp instance.
	 */
	AddToDBAppMgrHelper( LoginApp & loginApp ) :
		AddToManagerHelper( loginApp.mainDispatcher() ),
		app_( loginApp )
	{
		// Auto-send on construction.
		this->send();
	}


	/* Override from AddToManagerHelper. */
	void handleFatalTimeout() /* override */
	{
		ERROR_MSG( "AddToDBAppMgrHelper::handleFatalTimeout: Unable to add "
				"LoginApp to DBAppMgr (%s). Terminating.\n",
			app_.dbAppMgr().addr().c_str() );
		app_.mainDispatcher().breakProcessing();
	}


	/* Override from AddToDBAppHelper. */
	void doSend() /* override */
	{
		Mercury::Bundle	& bundle = app_.dbAppMgr().bundle();
		bundle.startRequest( DBAppMgrInterface::addLoginApp, this );
		app_.dbAppMgr().send();
	}


	/* Override from AddToDBAppHelper. */
	bool finishInit( BinaryIStream & data ) /* override */
	{
		LoginAppID appID;
		Mercury::Address dbAppAlphaAddr;
		data >> appID >> dbAppAlphaAddr;
		return app_.finishInit( appID, dbAppAlphaAddr );
	}

private:
	LoginApp & app_;
};

这里的doSend会往LoginApp::dbAppMgr发送消息，但是发送消息的前提是知道发送目标的ip:port，这个dbAppMgr的地址获取是在LoginApp::init中完成的:

bool LoginApp::init( int argc, char * argv[] ) /* override */
{
	// 省略一些代码
	int numStartupRetries = Config::numStartupRetries();

	if (!BW_INIT_ANONYMOUS_CHANNEL_CLIENT( dbAppMgr_, this->intInterface(),
			LoginIntInterface, DBAppMgrInterface, numStartupRetries ))
	{
		ERROR_MSG( "LoginApp::init: Could not find DBAppMgr\n" );
		return false;
	}
	// 省略一些代码
}

这个宏展开之后会调用下面的AnonymousChannelClient::init方法，内部会调用MachineDaemon::findInterface来查找DBAppMgr的地址:

/**
 *	This method initialises this object.
 *
 *	@return true on success, otherwise false.
 */
bool AnonymousChannelClient::init( Mercury::NetworkInterface & interface,
		Mercury::InterfaceMinder & interfaceMinder,
		const Mercury::InterfaceElement & birthMessage,
		const char * componentName,
		int numRetries )
{
	interfaceName_ = componentName;

	bool result = true;

	interfaceMinder.handler( birthMessage.id(), this );

	if (Mercury::MachineDaemon::registerBirthListener(
				interface.address(), birthMessage, componentName ) !=
			Mercury::REASON_SUCCESS)
	{
		NETWORK_ERROR_MSG( "AnonymousChannelClient::init: "
			"Failed to register birth listener for %s\n",
			componentName );

		result = false;
	}

	Mercury::Address serverAddr( Mercury::Address::NONE );

	if (Mercury::MachineDaemon::findInterface(
			componentName, 0, serverAddr, numRetries ) !=
		Mercury::REASON_SUCCESS)
	{
		result = false;
	}

	// Everyone talking to another process via this mechanism is doing it
	// irregularly at the moment.  Could make this optional.
	pChannelOwner_ = new Mercury::ChannelOwner( interface, serverAddr );
	pChannelOwner_->channel().isLocalRegular( false );
	pChannelOwner_->channel().isRemoteRegular( false );

	return result;
}

如果LoginApp启动的时候对应的DBAppMgr进程还没有启动，上面注册的BirthListener就会起作用，因为其会监听广播出来的DBAppMgrInterfaceBirth消息。

/**
 *	This method is used to register a birth or death listener with machined.
 */
Reason registerListener( const Address & srcAddr,
		UDPBundle & bundle, int addrStart,
		const char * ifname, bool isBirth, bool anyUID = false )
{
	// finalise the bundle first
	bundle.finalise();
	const Packet * p = bundle.pFirstPacket();

	MF_ASSERT( p->flags() == 0 );

	// prepare the message for machined
	ListenerMessage lm;
	lm.param_ = (isBirth ? lm.ADD_BIRTH_LISTENER : lm.ADD_DEATH_LISTENER) |
		lm.PARAM_IS_MSGTYPE;
	lm.category_ = lm.SERVER_COMPONENT;
	lm.uid_ = anyUID ? lm.ANY_UID : getUserId();
	lm.pid_ = mf_getpid();
	lm.port_ = srcAddr.port;
	lm.name_ = ifname;

	const int addrLen = 6;
	unsigned int postSize = p->totalSize() - addrStart - addrLen;

	lm.preAddr_ = BW::string( p->data(), addrStart );
	lm.postAddr_ = BW::string( p->data() + addrStart + addrLen, postSize );

	const uint32 destAddr = LOCALHOST;
	return lm.sendAndRecv( srcAddr.ip, destAddr, NULL );
}


/**
 *  This method registers a callback with machined to be called when a certain
 *	type of process is started.
 *
 *	@note This needs to be fixed up if rebind is called on this nub.
 */
Reason registerBirthListener( const Address & srcAddr,
		UDPBundle & bundle, int addrStart, const char * ifname )
{
	return registerListener( srcAddr, bundle, addrStart, ifname, true );
}

这个ListenerMessage的发送目标还是LOCALHOST，当LOCALHOST上的BWMachineD接收到这个消息的时候，会先存储这个Listener到内部的birthListeners_数组里:

bool BWMachined::handleMessage( Endpoint & ep, sockaddr_in & sin,
	MachineGuardMessage & mgm, MGMPacket & replies )
{
	// If the message we received is not known, don't attempt to process
	// any further. It is possible the data received is a valid structure
	// that has had its tail corrupted.
	if (mgm.flags_ & mgm.MESSAGE_NOT_UNDERSTOOD)
	{
		syslog( LOG_ERR, "Received unknown message: %s", mgm.c_str() );
		mgm.outgoing( true );
		replies.append( mgm );
		return true;
	}

	switch (mgm.message_)
	{

	case MachineGuardMessage::LISTENER_MESSAGE:
	{
		ListenerMessage &lm = static_cast< ListenerMessage& >( mgm );

		if (lm.param_ == (lm.ADD_BIRTH_LISTENER | lm.PARAM_IS_MSGTYPE))
		{
			birthListeners_.add( lm, sin.sin_addr.s_addr );
		}
		else if (lm.param_ == (lm.ADD_DEATH_LISTENER | lm.PARAM_IS_MSGTYPE))
		{
			deathListeners_.add( lm, sin.sin_addr.s_addr );
		}
		else
		{
			syslog( LOG_ERR, "Unrecognised param field for ListenerMessage: %x",
				lm.param_ );
			return false;
		}

		// Ack to sender
		lm.outgoing( true );
		replies.append( lm );

		return true;
	}
	// 省略其他代码
	}
}

但是DBAppMgr可能是在其他机器上启动的，所以肯定有某种机制在DBAppMgr启动之后，将其ip:port广播出来,为了跟踪这个广播过程，我们需要查看使用这个BirthListener的地方，依然在BWMachined::handleMessage方法中，在处理ProcessMessage::NOTIFY_BIRTH消息的分支里:

case MachineGuardMessage::PROCESS_MESSAGE:
{
	ProcessMessage &pm = static_cast< ProcessMessage& >( mgm );

	// Explicit instances of this class shouldn't be using param filters
	if (!(pm.param_ & pm.PARAM_IS_MSGTYPE))
	{
		syslog( LOG_ERR, "PROCESS_MESSAGE tried to use param filters! (%x)",
			pm.param_ );
		return false;
	}

	int msgtype = pm.param_ & ~pm.PARAM_IS_MSGTYPE;

	// Don't allow other machines to register/deregister their processes
	// on this machine.
	if ((msgtype == pm.REGISTER || msgtype == pm.DEREGISTER) &&
		(uint32&)sin.sin_addr != cluster_.ownAddr_ &&
		(uint32&)sin.sin_addr != LOCALHOST)
	{
		syslog( LOG_ERR, "%s tried to register a process here!",
			inet_ntoa( sin.sin_addr ) );
		return false;
	}

	switch (msgtype)
	{
	case ProcessMessage::REGISTER:
	{
		// 省略相关代码
	}

	case ProcessMessage::DEREGISTER:
	{
		// 省略相关代码
	}

	case ProcessMessage::NOTIFY_BIRTH:
	{
		birthListeners_.handleNotify( this->endpoint(), pm, sin.sin_addr );
		return true;
	}

	case ProcessMessage::NOTIFY_DEATH:
	{
		deathListeners_.handleNotify( this->endpoint(), pm, sin.sin_addr );
		return true;
	}

	default:
		syslog( LOG_ERR, "Unrecognised ProcessMessage type: %d", msgtype );
		return false;
	}
}

这里接收到ProcessMessage::NOTIFY_BIRTH类型的消息的时候会调用Listeners::handleNotify方法，这个方法会遍历内部的birthListeners_数组，将消息发送给所有注册的BirthListener：

void Listeners::handleNotify( const Endpoint & endpoint,
	const ProcessMessage & pm, in_addr addr )
{
	char address[6];
	memcpy( address, &addr, sizeof( addr ) );
	memcpy( address + sizeof( addr ), &pm.port_, sizeof( pm.port_ ) );

	Members::iterator iter = members_.begin();

	while (iter != members_.end())
	{
		ListenerMessage &lm = iter->lm_;

		if (lm.category_ == pm.category_ &&
			(lm.uid_ == lm.ANY_UID || lm.uid_ == pm.uid_) &&
			(lm.name_ == pm.name_ || lm.name_.size() == 0))
		{
			int msglen = lm.preAddr_.size() + sizeof( address ) +
				lm.postAddr_.size();
			char *data = new char[ msglen ];
			int preSize = lm.preAddr_.size();
			int postSize = lm.postAddr_.size();

			memcpy( data, lm.preAddr_.c_str(), preSize );
			memcpy( data + preSize, address, sizeof( address ) );
			memcpy( data + preSize + sizeof( address ), lm.postAddr_.c_str(),
				postSize );

			// and send to the appropriate port locally
			endpoint.sendto( data, msglen, lm.port_, iter->addr_ );
			delete [] data;
		}

		++iter;
	}
}

因此需要继续跟踪一下这个ProcessMessage::NOTIFY_BIRTH消息又是在哪里发出来的，还是在上面的MachineGuardMessage::PROCESS_MESSAGE分支里,不过是在处理ProcessMessage::REGISTER消息的时候，这里会使用broadcastToListeners执行NOTIFY_BIRTH消息的广播，这个broadcastToListeners的发送目标是广播地址BROADCAST，因此任何本地局域网里的机器上的BWMachineD都可以接收到这个NOTIFY_BIRTH的消息:

switch (msgtype)
{
case ProcessMessage::REGISTER:
{
	// 省略一些代码

	// platform-specific initialisation
	pi.init( pm );

	// Tell listeners about it
	broadcastToListeners( pm, pm.NOTIFY_BIRTH );

	// and confirm the registration to the sender
	pm.outgoing( true );
	replies.append( pm );

	syslog( LOG_INFO, "Added %s for uid:%d (debug: at index %d)\n",
		pm.c_str(), pm.uid_, i );
	return true;
}
// 省略一些代码
}

/**
 * Inform other machined's on the network about the birth or death of a process
 * on this machine so that they can in turn inform their registered listeners
 * about it.
 */
bool BWMachined::broadcastToListeners( ProcessMessage &pm, int type )
{
	uint8 oldparam = pm.param_;
	pm.param_ = type | pm.PARAM_IS_MSGTYPE;

	bool ok = pm.sendto( ep_, htons( PORT_MACHINED ), BROADCAST,
		MGMPacket::PACKET_STAGGER_REPLIES );

	pm.param_ = oldparam;
	return ok;
}

跟踪完整个BirthListener的调用链，可以知道当DBAppMgr进程启动之后，会向本地的BWMachineD发送ProcessMessage::REGISTER消息，本地的BWMachineD就会以ProcessMessage::NOTIFY_BIRTH类型的消息广播给所有局域网机器上注册的BirthListener，包括LoginApp，因此LoginApp就会收到BirthListener的回调，从而完成DBAppMgr的地址获取:

/**
 *	This method handles a birth message telling us that a new server has
 *	started.
 */
void AnonymousChannelClient::handleMessage(
		const Mercury::Address & srcAddr,
		Mercury::UnpackedMessageHeader & header,
		BinaryIStream & data )
{
	Mercury::Address serverAddr;
	data >> serverAddr;

	MF_ASSERT( data.remainingLength() == 0 && !data.error() );

	pChannelOwner_->addr( serverAddr );

	PROC_IP_INFO_MSG( "AnonymousChannelClient::handleMessage: "
		"Got new %s at %s\n",
		interfaceName_.c_str(), pChannelOwner_->channel().c_str() );
}

退出流程

基于信号的退出

Bigworld里的进程都注册了强制退出信号处理函数，目前处理的退出信号是SIGINT，在ServerApp的初始化函数init里会调用enableSignalHandler方法注册这个SIGINT信号处理函数到自身的pSignalHandler_:

bool ServerApp::init( int argc, char * argv[] )
{

	// 省略一些代码

	pSignalHandler_.reset( this->createSignalHandler() );

	// Handle signals
	this->enableSignalHandler( SIGINT );

	return true;
}

SignalHandler * ServerApp::createSignalHandler()
{
	return new ServerAppSignalHandler( *this );
}
/**
 *	Enables or disables the handling of a given signal by the ServerApp
 *	instance's signal handler.
 */
void ServerApp::enableSignalHandler( int sigNum, bool enable )
{
	if (pSignalHandler_.get() == NULL)
	{
		ERROR_MSG( "ServerApp::enableSignalHandler: no signal handler set\n" );
		return;
	}

	if (enable)
	{
		SignalProcessor::instance().addSignalHandler( sigNum, 
			pSignalHandler_.get() );
	}
	else
	{
		SignalProcessor::instance().clearSignalHandler( sigNum, 
			pSignalHandler_.get() );
	}
}

这个pSignalHandler_负责将信号处理函数转发到ServerApp::onSignalled方法，这里会利用signal( sigNum, SIG_DFL )重置为默认的信号处理函数，来避免Python脚本那边设置了这个信号的处理，然后使用kill( getpid(), sigNum )来重新发送这个信号，从而导致进程的强制退出:

class ServerAppSignalHandler : public SignalHandler
{
public:
	ServerAppSignalHandler( ServerApp & serverApp ):
			serverApp_( serverApp )
	{}

	virtual ~ServerAppSignalHandler()
	{}

	virtual void handleSignal( int sigNum )
	{
		serverApp_.onSignalled( sigNum );
	}
private:
	ServerApp & serverApp_;
};

/**
 *	Default signal handling.
 */
void ServerApp::onSignalled( int sigNum )
{
	switch (sigNum)
	{
	case SIGINT:
		// This is really just to make sure that Python does not install its
		// own SIGINT handler.
		// It would probably be better to just revert to SIG_DFL straight after
		// Py_Initialize in Script::init().
		signal( sigNum, SIG_DFL );
		kill( getpid(), sigNum );
		break;

	default:
		break;
	}

}

基于shutdown的退出

上述的强制kill会导致进程立即退出，缺少很多资源的释放与清理操作，所以一般不在迫不得已的情况下不会去使用这个退出信号。在ServerApp上提供了一个更加优雅的退出接口ServerApp::shutDown，ServerApp::shutDown方法里会调用mainDispatcher_.breakProcessing方法来通知事件循环处理processContinuously可以结束了:

void ServerApp::shutDown()
{
	INFO_MSG( "ServerApp::shutDown: shutting down\n" );
	mainDispatcher_.breakProcessing();
}
/**
 *	This method breaks out of 'processContinuously' at the next opportunity.
 *	Any messages in bundles that are being processed or timers that have
 *	expired will still get called. Note: if this is called from another
 *	thread it will NOT interrupt a select call if one is in progress, so
 *	processContinuously will not return. Try sending the process a (handled)
 *	signal if this is your intention.
 */
INLINE void EventDispatcher::breakProcessing( bool breakState )
{
	breakProcessing_ = breakState;
}

void EventDispatcher::processContinuously()
{
	breakProcessing_ = false;

	while (!breakProcessing_)
	{
		this->processOnce( /* shouldIdle */ true );
	}
}

当这个processContinuously方法返回时，ServerApp就会执行onRunComplete方法，开始执行清理操作:

/**
 *	This is the default implementation of run. Derived classes to override
 *	this to implement their own functionality.
 */
bool ServerApp::run()
{
	mainDispatcher_.processUntilBreak();

	this->onRunComplete();

	return true;
}

这个onRunComplete是一个虚函数，目前在ServerApp里的实现是空，在子类里会根据需要去重载:

/*
 *	Override from ServerApp.
 */
void LoginApp::onRunComplete() /* override */
{
	INFO_MSG( "LoginApp::run: Terminating normally.\n" );
	this->ServerApp::onRunComplete();
	bool sent = false;

	if (this->isDBAppMgrReady())
	{
		Mercury::Bundle	& dbMgrBundle = dbAppMgr_.pChannelOwner()->bundle();
		DBAppMgrInterface::controlledShutDownArgs args;
		args.stage = SHUTDOWN_REQUEST;
		dbMgrBundle << args;
		dbAppMgr_.pChannelOwner()->send();
		sent = true;
	}

	if (sent)
	{
		this->intInterface().processUntilChannelsEmpty();
	}
}
/**
 *  This method is run when the runloop has finished.
 */
void DBApp::onRunComplete()
{
    this->ScriptApp::onRunComplete();
    this->finalise();
}

在ServerApp::run返回之后，会接下来执行fini方法来清理，并在fini结束之后调用interface_.prepareForShutdown方法来准备网络接口的关闭:

bool ServerApp::runApp( int argc, char * argv[] )
{
	// calculate the clock speed
	stampsPerSecond();

	bool result = false;

	if (this->init( argc, argv ))
	{
		INFO_MSG( "---- %s is running ----\n", this->getAppName() );
		result = this->run();
	}
	else
	{
		ERROR_MSG( "Failed to initialise %s\n", this->getAppName() );
	}

	this->fini();

	interface_.prepareForShutdown();

#if ENABLE_PROFILER
	g_profiler.fini();
#endif

	return result;
}

这个fini方法也是一个声明在ServerApp里的空的虚函数，在子类里会根据需要去重载，用来执行一些清理操作。不过目前好像没有在哪个ServerApp的子类里看到重载这个方法的情况。

至于interface_.prepareForShutdown方法，主要任务是用来调用mainDispatcher.prepareforshutdown，然后通过finaliseRequestManager方法来清理请求管理器,这样就不再处理任何网络请求:

/**
 *	This method prepares this NetworkInterface for being shut down.
 */
void NetworkInterface::prepareForShutdown()
{
	if (pMainDispatcher_)
	{
		pMainDispatcher_->prepareForShutdown();
	}

	this->finaliseRequestManager();
}

/**
 *	This method destroys the RequestManager. It should be called when shutting
 *	down to ensure that no ReplyMessageHandler instances are cancelled when the
 *	server is in a bad state (e.g. after the App has been destroyed.)
 */
void NetworkInterface::finaliseRequestManager()
{
	this->interfaceTable().serve( InterfaceElement::REPLY, NULL );

	RequestManager * pRequestManager = pRequestManager_;
	pRequestManager_ = NULL;
	bw_safe_delete( pRequestManager );
}

在EventDispatcher::prepareForShutdown方法中，会重新使用循环来调用processOnce方法，处理退出阶段的所有逻辑，这里设置了处理的最大超时时间为2s，避免业务逻辑出错引发退出失败:

void EventDispatcher::prepareForShutdown()
{
	const uint64 SECONDS_TO_ATTEMPT_SEND = 2;

	const uint64 startTime = timestamp();
	uint64 timePeriod = (stampsPerSecond() * SECONDS_TO_ATTEMPT_SEND);

	while (!pPoller_->isReadyForShutdown() &&
		(timestamp() - startTime < timePeriod))
	{
		this->processOnce();

		// Wait 100ms
#if defined( PLAYSTATION3 )
		sys_timer_usleep( 100000 );
#elif !defined( _WIN32 )
		usleep( 100000 );
#else
		Sleep( 100 );
#endif
	}
}

当这些流程都走完之后，就开始调用各个ServerApp的析构函数，虽然这个基类上的析构函数是一个空实现，各个子类里的重载内容就十分丰富了，简单点的就是LoginApp，在析构函数里关闭对外开放的网络端口extInterface:

/**
 *	Destructor.
 */
LoginApp::~LoginApp()
{
	this->extInterface().prepareForShutdown();
	statsTimer_.cancel();
	tickTimer_.cancel();
}

在DBApp的析构函数里会执行数据库的断开操作:

/**
 *  Destructor.
 */
DBApp::~DBApp()
{
    interface_.cancelRequestsFor( &baseAppMgr_.channel() );

    statusCheckTimer_.cancel();
    gameTimer_.cancel();

    bw_safe_delete( pBillingSystem_ );
    pBillingSystem_ = NULL;

    bw_safe_delete( pDatabase_ );
    // Destroy entity descriptions before calling Script::fini() so that it
    // can clean up any PyObjects that it may have.
    bw_safe_delete( pEntityDefs_ );
    DataType::clearStaticsForReload();

    // Now done in ~ScriptApp
    // Script::fini();
}

这样就完成了有序退出shutDown流程。

至于这个shutDown接口什么时候被调用，则根据具体的ServerApp子类来管理，最简单的就是将这个接口暴露给Python，让业务逻辑来决定什么时候当前进程应该退出，例如在CellApp和BaseApp里就提供了shutDownApp方法给Python调用:

// server\baseapp\script_bigworld.cpp
/*~ function BigWorld.shutDownApp
 *	@components{ base }
 *
 *	This method induce a shutdown of this BaseApp.
 */
void shutDownApp()
{
	BaseApp::instance().shutDown();
}
PY_AUTO_MODULE_FUNCTION( RETVOID, shutDownApp, END, BigWorld )

// server\cellapp\cellapp.cpp
/*~ function BigWorld.shutDownApp
 *	@components{ cell }
 *
 *	This method induce a controlled shutdown of this CellApp.
 */
void shutDownApp()
{
	CellApp::instance().shutDown();
}
PY_AUTO_MODULE_FUNCTION( RETVOID, shutDownApp, END, BigWorld )

基于retire的退出

如果要一个ServerApp执行安全退出，上述的shutDownApp是一个非常方便的方法。此外还有一种方法可以触发ServerApp::shutDown方法来执行安全的退出逻辑，这个方法是通过BWMachineD发送的retireApp指令。每个ServerApp在启动时都会通过ServerApp::addWatchers注册一个command/retireXXX指令的监听，这里的XXX是当前进程的在集群里的名字AppName：当BWMachineD发送retireApp指令时，ServerApp会收到这个指令，然后执行安全退出逻辑。

/**
 *	This method adds the watchers associated with this class.
 */
void ServerApp::addWatchers( Watcher & watcher )
{
	watcher.addChild( "nub",
			Mercury::NetworkInterface::pWatcher(), &interface_ );

	if (this->pManagerAppGateway())
	{
		char buf[ 256 ];
		snprintf( buf, sizeof( buf ), "command/retire%s",
				this->getAppName() );

		watcher.addChild( buf, new RetireAppCommand( *this ) );
	}
}
class RetireAppCommand : public NoArgCallableWatcher
{
public:
	RetireAppCommand( ServerApp & app ) :
		// TODO: BWT-29273 Enable DBApp watcher forwarding 
		NoArgCallableWatcher( strcmp( app.getAppName(), "DBApp" ) == 0 ?
					CallableWatcher::LOCAL_ONLY : CallableWatcher::LEAST_LOADED,
				"Retire the least loaded app." ),
		app_( app )
	{
	}

private:
	virtual bool onCall( BW::string & output, BW::string & value )
	{
		INFO_MSG( "Requesting to retire this app.\n" );
		app_.requestRetirement();
		return true;
	}

	ServerApp & app_;
};

当ServerApp接收到了command/retireXXX指令后，会调用ServerApp::requestRetirement方法来请求退休。在ServerApp::requestRetirement方法里，会判断当前进程是否有ManagerAppGateway，如果有，就会调用ManagerAppGateway::retireApp方法来发送退休请求。

/**
 *	This method requests that this application should be retired.
 */
void ServerApp::requestRetirement()
{
	if (!this->pManagerAppGateway())
	{
		ERROR_MSG( "ServerApp::requestRetirement: "
				"%s has no manager app gateway configured\n",
			this->getAppName() );
		return;
	}

	this->pManagerAppGateway()->retireApp();
}
void ManagerAppGateway::retireApp()
{
	Mercury::Bundle & bundle = channel_.bundle();
	bundle.startMessage( retireAppIE_ );
	channel_.send();
}

这个retireAppIE_是在ManagerAppGateway初始化的时候就设置好了的，CellAppMgrGateway、BaseAppMgrGateway和DBAppMgrGateway都是ManagerAppGateway的子类，在构造函数里就初始化好了retireAppIE_。

ManagerAppGateway::ManagerAppGateway( Mercury::NetworkInterface & networkInterface,
			const Mercury::InterfaceElement & retireAppIE ) :
		channel_( networkInterface, Mercury::Address::NONE ),
		retireAppIE_( retireAppIE )
{
	MF_ASSERT( retireAppIE_.lengthStyle() == Mercury::FIXED_LENGTH_MESSAGE );
	MF_ASSERT( retireAppIE_.lengthParam() == 0 );
}

/**
 *	The constructor for CellAppMgrGateway.
 */
CellAppMgrGateway::CellAppMgrGateway( Mercury::NetworkInterface & interface ) :
	ManagerAppGateway( interface, CellAppMgrInterface::retireApp )
{}

BaseAppMgrGateway::BaseAppMgrGateway( Mercury::NetworkInterface & interface ) :
		ManagerAppGateway( interface, BaseAppMgrInterface::retireApp )
{
}
/**
 *	The constructor for DBAppMgrGateway.
 */
DBAppMgrGateway::DBAppMgrGateway( Mercury::NetworkInterface & interface ) :
	ManagerAppGateway( interface, DBAppMgrInterface::retireApp )
{}

这三个AppMgr处理retireApp指令的方式基本一样，这里只分析CellAppMgr里对应的处理逻辑，这个retireApp的指令会发送到CellAppMgr里创建的对应的CellApp对象:

/**
 *	This method is called by the cell application when it wants to be shut down.
 */
void CellApp::retireApp()
{
	INFO_MSG( "CellApp::retireApp: CellApp %u is requesting to retire.\n",
			this->id() );
	this->startRetiring();
}
/**
 *	This method starts the process of shutting down the application
 *	associated with this object.
 */
void CellApp::startRetiring()
{
	INFO_MSG( "CellApp::startRetiring: Retiring CellApp %u - %s\n",
			id_, this->addr().c_str() );

	isRetiring_ = true;

	if (!cells_.empty())
	{
		this->retireAllCells();
	}
	else
	{
		this->shutDown();
	}
}

/**
 *	This method shuts down the cell application associated with this object.
 */
void CellApp::shutDown()
{
	CellAppInterface::shutDownArgs args;
	args.isSigInt = false; // Not used.

	this->bundle() << args;
	this->send();
}

CellApp接收到这个retireApp指令后，会调用CellApp::startRetiring方法来启动退休过程。在CellApp::startRetiring方法里，会判断当前CellApp是否管理着cells（游戏区域），如果有，就会调用CellApp::retireAllCells方法来迁移cells到其他CellApp，如果没有，就会调用CellApp::shutDown方法来退出进程。

当CellApp管理的cells都迁移到其他CellApp后，如果发现当前已经设置好了retire标记位，则也会调用CellApp::shutDown方法来发送RPC通知对应的CellApp进程去退出:

/**
 *	This method starts retiring all cells from this app.
 */
void CellApp::retireAllCells()
{
	cells_.retireAll();
}

/**
 *	This method erases the input cell from this application.
 */
void CellApp::eraseCell( CellData * pCellData )
{
	cells_.erase( pCellData );

	if (isRetiring_ && cells_.empty())
	{
		this->shutDown();
	}
}

当CellApp进程收到shutDown指令后，会调用CellApp::shutDown方法来退出安全退出:

/**
 *	This method handles a shutDown message.
 */
void CellApp::shutDown( const CellAppInterface::shutDownArgs & /*args*/ )
{
	TRACE_MSG( "CellApp::shutDown: Told to shut down\n" );
	this->shutDown();
}

集群的有序退出

上面暴露的shutDownApp方法只会触发当前进程的退出，如果要当前服务器集群里的所有进程都有序退出，一个一个的去通知shutDownApp不太灵活，因此在CellApp和BaseApp这两个能绑定Python的ServerApp子类里，提供了一个controlledShutDown方法，用来触发有序退出流程:

// server\baseapp\script_bigworld.cpp
/*~ function BigWorld.controlledShutDown
 *	@components{ base }
 *
 *	This method induces a controlled shutdown of the cluster.
 */
void controlledShutDown()
{
	BaseApp::instance().triggerControlledShutDown();
}
PY_AUTO_MODULE_FUNCTION( RETVOID, controlledShutDown, END, BigWorld )

// server\cellapp\cellapp.cpp
/*~ function BigWorld.controlledShutDown
 *	@components{ cell }
 *
 *	This method induces a controlled shutdown of the cluster.
 */
void controlledShutDown()
{
	CellApp::instance().triggerControlledShutDown();
}
PY_AUTO_MODULE_FUNCTION( RETVOID, controlledShutDown, END, BigWorld )

这两个不同的ServerApp执行controlledShutDown方法的后续流程大同小异。这里先分析一下CellApp的triggerControlledShutDown方法:

/**
 *	This method triggers a controlled shutdown of the cluster.
 */
void CellApp::triggerControlledShutDown()
{
	CellAppMgrGateway & cellAppMgr = this->cellAppMgr();
	Mercury::Bundle & bundle = cellAppMgr.bundle();

	CellAppMgrInterface::controlledShutDownArgs &args =
		CellAppMgrInterface::controlledShutDownArgs::start( bundle );

	args.stage = SHUTDOWN_TRIGGER;

	cellAppMgr.send();
}

这个函数会向CellAppMgr发送一个controlledShutDown消息，携带的stage参数为SHUTDOWN_TRIGGER，CellAppMgr收到这个消息之后,会执行自己的triggerControlledShutDown方法:

/**
 *	This method handles a message to start shutting down in a controlled way.
 */
void CellAppMgr::controlledShutDown(
			const CellAppMgrInterface::controlledShutDownArgs & args )
{
	INFO_MSG( "CellAppMgr::controlledShutDown: stage = %s\n", 
		ServerApp::shutDownStageToString( args.stage ) );

	switch (args.stage)
	{
		case SHUTDOWN_TRIGGER:
			this->triggerControlledShutDown();
			break;
s
		case SHUTDOWN_REQUEST:
			this->controlledShutDown();
			break;

		default:
			ERROR_MSG( "CellAppMgr::controlledShutDown: "
					"Stage %s not handled.\n", 
				ServerApp::shutDownStageToString( args.stage ) );
			break;
	}
}

这个CellAppMgr::triggerControlledShutDown负责将SHUTDOWN_TRIGGER消息发送给BaseAppMgr:

/**
 *	This method triggers a controlled shutdown of the cluster.
 */
void CellAppMgr::triggerControlledShutDown()
{
	Mercury::Bundle & bundle = baseAppMgr_.bundle();
	BaseAppMgrInterface::controlledShutDownArgs &args =
		BaseAppMgrInterface::controlledShutDownArgs::start( bundle );
	args.stage = SHUTDOWN_TRIGGER;
	args.shutDownTime = 0; // unused on receiving side

	baseAppMgr_.send();
}

BaseAppMgr收到这个消息之后,会执行自己的triggerControlledShutDown方法:

/**
 *	This method responds to a message telling us what stage of the controlled
 *	shutdown process the server is at.
 */
void BaseAppMgr::controlledShutDown(
		const BaseAppMgrInterface::controlledShutDownArgs & args )
{
	INFO_MSG( "BaseAppMgr::controlledShutDown: stage = %s\n", 
		ServerApp::shutDownStageToString( args.stage ) );

	switch (args.stage)
	{
		// 省略一些代码

		case SHUTDOWN_TRIGGER:
		{
			this->controlledShutDownServer();
			break;
		}

		case SHUTDOWN_NONE:
		case SHUTDOWN_DISCONNECT_PROXIES:
			break;
	}
}

类似的，如果是BaseApp发起的triggerControlledShutDown方法，会向BaseAppMgr发送一个controlledShutDown消息，携带的stage参数为SHUTDOWN_TRIGGER，此时也会调用到BaseAppMgr::controlledShutDownServer方法：

/**
 *	This method triggers a controlled shutdown of the cluster.
 */
void BaseApp::triggerControlledShutDown()
{
	BaseAppMgrGateway & baseAppMgr = this->baseAppMgr();

	Mercury::Bundle & bundle = baseAppMgr.bundle();
	BaseAppMgrInterface::controlledShutDownArgs &args =
		BaseAppMgrInterface::controlledShutDownArgs::start( bundle );

	args.stage = SHUTDOWN_TRIGGER;
	args.shutDownTime = 0; // unused on receiving side

	baseAppMgr.send();
}

所以不管是CellApp还是BaseApp发起的triggerControlledShutDown方法，最终都会调用到BaseAppMgr::controlledShutDownServer方法。在BaseAppMgr::controlledShutDownServer里，会将controlledShutDown请求发送到DbAppMgr和LoginApp上，然后以SHUTDOWN_REQUEST消息的形式重新执行controlledShutDown方法:

/**
 *  Trigger a controlled shutdown of the entire server.
 */
void BaseAppMgr::controlledShutDownServer()
{
	if (shutDownStage_ != SHUTDOWN_NONE)
	{
		DEBUG_MSG( "BaseAppMgr::controlledShutDownServer: "
			"Already shutting down, ignoring duplicate shutdown request.\n" );
		return;
	}

	// First try to trigger controlled shutdown via the loginapp
	Mercury::Address loginAppAddr;
	Mercury::Reason reason = Mercury::MachineDaemon::findInterface(
				"LoginIntInterface", -1, loginAppAddr );

	if (reason == Mercury::REASON_SUCCESS)
	{
		Mercury::ChannelSender sender( BaseAppMgr::getChannel( loginAppAddr ) );
		Mercury::Bundle & bundle = sender.bundle();

		bundle.startMessage( LoginIntInterface::controlledShutDown );

		INFO_MSG( "BaseAppMgr::controlledShutDownServer: "
			"Triggering server shutdown via LoginApp @ %s\n",
			loginAppAddr.c_str() );

		return;
	}
	else
	{
		ERROR_MSG( "BaseAppMgr::controlledShutDownServer: "
			"Couldn't find a LoginApp to trigger server shutdown\n" );
	}

	// Next try to trigger shutdown via the DBApp.
	// TODO: Scalable DB. Talk directly to DBAppMgr, instead of DBApp. DBApp
	// will forward to DBAppMgr currently.
	if (dbAppAlpha_.channel().isEstablished())
	{
		Mercury::Bundle & bundle = dbAppAlpha_.bundle();
		DBAppInterface::controlledShutDownArgs::start( bundle ).stage =
			SHUTDOWN_REQUEST;
		dbAppAlpha_.send();

		INFO_MSG( "BaseAppMgr::controlledShutDownServer: "
				"Triggering server shutdown via the Alpha DBApp%02d (%s)\n",
			dbApps_.alpha().id(), 
			dbAppAlpha_.addr().c_str() );
		return;
	}
	else
	{
		ERROR_MSG( "BaseAppMgr::controlledShutDownServer: "
			"Couldn't find the DBApp to trigger server shutdown\n" );
	}

	// Alright, the shutdown starts with me then
	BaseAppMgrInterface::controlledShutDownArgs args;
	args.stage = SHUTDOWN_REQUEST;
	INFO_MSG( "BaseAppMgr::controlledShutDownServer: "
		"Starting controlled shutdown here (no LoginApp or DBApp found)\n" );
	this->controlledShutDown( args );
}

在CellAppMgr::controlledShutDown里，遇到SHUTDOWN_REQUEST消息时，会将这个controlledShutDown请求发送到CellAppMgr上。当CellAppMgr接收到这个SHUTDOWN_REQUEST消息时，会执行自己的controlledShutDown方法，开启自身的退出流程:

/**
 *	This method handles a message to start shutting down in a controlled way.
 */
void CellAppMgr::controlledShutDown(
			const CellAppMgrInterface::controlledShutDownArgs & args )
{
	INFO_MSG( "CellAppMgr::controlledShutDown: stage = %s\n", 
		ServerApp::shutDownStageToString( args.stage ) );

	switch (args.stage)
	{
		case SHUTDOWN_TRIGGER:
			this->triggerControlledShutDown();
			break;

		case SHUTDOWN_REQUEST:
			this->controlledShutDown();
			break;

		default:
			ERROR_MSG( "CellAppMgr::controlledShutDown: "
					"Stage %s not handled.\n", 
				ServerApp::shutDownStageToString( args.stage ) );
			break;
	}
}
/**
 *	This method initiates a controlled shutdown of the system.
 */
void CellAppMgr::controlledShutDown()
{
	// Stop sending to anonymous channels etc.
	interface_.stopPingingAnonymous();

	ShutDownHandler::start( *this );
}

这里的ShutDownHandler::start的作用是通知当前CellAppMgr管理的所有CellApp和BaseAppMgr开始执行退出流程controlledShutDown,此时的stage为SHUTDOWN_INFORM:

/**
 *	This method starts the process of shutting down the server.
 */
void ShutDownHandler::start( CellAppMgr & mgr )
{
	// No delay if we haven't started yet since game time won't go forward.
	int shuttingDownDelay = 0;

	if (mgr.hasStarted())
	{
		shuttingDownDelay = CellAppMgrConfig::shuttingDownDelayInTicks();
	}

	mgr.setShutDownHandler(
		new InformHandler( mgr, mgr.time() + shuttingDownDelay ) );
}

InformHandler::InformHandler( CellAppMgr & mgr, GameTime shutDownTime ) :
	mgr_( mgr ),
	shutDownTime_( shutDownTime ),
	baseAppsAcked_( false ),
	ackedCellApps_()
{
	// Inform the BaseAppMgr.
	{
		Mercury::Bundle & bundle = mgr_.baseAppMgr().bundle();
		BaseAppMgrInterface::controlledShutDownArgs args;
		args.stage = SHUTDOWN_INFORM;
		args.shutDownTime = shutDownTime_;
		bundle << args;

		mgr_.baseAppMgr().send();
	}

	// Inform all of the CellApps.
	mgr_.cellApps().controlledShutDown( SHUTDOWN_INFORM, shutDownTime_ );
}


/**
 *
 */
void CellApps::controlledShutDown( ShutDownStage stage, GameTime shutDownTime )
{
	Map::iterator iter = map_.begin();

	while (iter != map_.end())
	{
		iter->second->controlledShutDown( stage, shutDownTime );

		++iter;
	}
}
/**
 *	This method shuts down the cell application associated with this object in
 *	a controlled way.
 */
void CellApp::controlledShutDown( ShutDownStage stage, GameTime shutDownTime )
{
	CellAppInterface::controlledShutDownArgs args;
	args.stage = stage;
	args.shutDownTime = shutDownTime;

	this->bundle() << args;
	this->send();
}

在BaseAppMgr::controlledShutDown方法中，当stage为SHUTDOWN_INFORM时，会通知当前管理的所有BaseApp开启退出流程:

/**
 *	This method responds to a message telling us what stage of the controlled
 *	shutdown process the server is at.
 */
void BaseAppMgr::controlledShutDown(
		const BaseAppMgrInterface::controlledShutDownArgs & args )
{
	INFO_MSG( "BaseAppMgr::controlledShutDown: stage = %s\n", 
		ServerApp::shutDownStageToString( args.stage ) );

	switch (args.stage)
	{
		case SHUTDOWN_REQUEST:
		{
			if (cellAppMgr_.channel().isEstablished())
			{
				Mercury::Bundle & bundle = cellAppMgr_.bundle();
				CellAppMgrInterface::controlledShutDownArgs args;
				args.stage = SHUTDOWN_REQUEST;
				bundle << args;
				cellAppMgr_.send();
			}
			break;
		}

		case SHUTDOWN_INFORM:
		{
			shutDownStage_ = args.stage;
			shutDownTime_ = args.shutDownTime;

			if (baseAndServiceApps_.empty())
			{
				this->ackBaseAppsShutDownToCellAppMgr( args.stage );
			}
			else
			{
				// Inform all base apps, once the requests are complete the
				// CellAppMgr is notified via ackBaseAppsShutDownToCellAppMgr().
				this->informBaseAppsOfShutDown( args );
			}

			break;
		}

		case SHUTDOWN_PERFORM:
		{
			this->startAsyncShutDownStage( SHUTDOWN_DISCONNECT_PROXIES );
			break;
		}

		case SHUTDOWN_TRIGGER:
		{
			this->controlledShutDownServer();
			break;
		}

		case SHUTDOWN_NONE:
		case SHUTDOWN_DISCONNECT_PROXIES:
			break;
	}
}

/**
 *	This method informs base and service apps of the controlled shutdown process.
 *	@param args Shutdown stage and time.
 */
void BaseAppMgr::informBaseAppsOfShutDown(
	const BaseAppMgrInterface::controlledShutDownArgs & args )
{
	MF_ASSERT( !baseAndServiceApps_.empty() );
	SyncControlledShutDownHandler * pHandler = new SyncControlledShutDownHandler(
		args.stage, baseAndServiceApps_.size() );
	MemoryOStream payload;
	payload << args.stage << args.shutDownTime;
	sendToBaseApps( BaseAppIntInterface::controlledShutDown, payload,
		BaseAppsIterator( baseAndServiceApps_ ), pHandler );
}

此时BaseAppMgr和CellAppMgr的shutDownStage_都被设置为SHUTDOWN_INFORM，同时所有的CellApp和BaseApp都接收到了controlledShutDown方法，且stage为SHUTDOWN_INFORM。

对应的处理逻辑都类似，记录当前的shutDownTime_和shutDownReplyID_，并触发onAppShuttingDown事件:

/**
 *	This method is called by the BaseAppMgr to tell us to shut down.
 */
void BaseApp::controlledShutDown( const Mercury::Address& srcAddr,
		const Mercury::UnpackedMessageHeader& header,
		BinaryIStream & data )
{
	ShutDownStage stage;
	GameTime shutDownTime;
	data >> stage >> shutDownTime;

	INFO_MSG( "BaseApp::controlledShutDown: stage = %s\n", 
		ServerApp::shutDownStageToString( stage ) );

	switch (stage)
	{
		case SHUTDOWN_INFORM:
		{
			// Make sure that we no longer process the external nub.
			extInterface_.detach();

			shutDownTime_ = shutDownTime;
			shutDownReplyID_ = header.replyID;

			if (this->hasStarted())
			{
				this->scriptEvents().triggerTwoEvents( "onAppShuttingDown",
						isServiceApp_ ? 
							"onServiceAppShuttingDown" :
							"onBaseAppShuttingDown",
						Py_BuildValue( "(d)",
							(double)shutDownTime/Config::updateHertz() ) );

				// Don't send reply immediate to allow scripts to do some stuff.
			}
			else
			{
				baseAppMgr_.bundle().startReply( shutDownReplyID_ );
				baseAppMgr_.send();
			}
		}
		// 省略其他分支的处理
	}
}

/**
 *	This method handles a message telling us to shut down in a controlled way.
 */
void CellApp::controlledShutDown(
		const CellAppInterface::controlledShutDownArgs & args )
{
	switch (args.stage)
	{
		case SHUTDOWN_INFORM:
		{
			if (shutDownTime_ != 0)
			{
				ERROR_MSG( "CellApp::controlledShutDown: "
						"Setting shutDownTime_ to %u when already set to %u\n",
					args.shutDownTime, shutDownTime_ );
			}

			shutDownTime_ = args.shutDownTime;

			hasAckedCellAppMgrShutDown_ = false;

			if (this->hasStarted())
			{
				this->scriptEvents().triggerTwoEvents(
					"onAppShuttingDown", "onCellAppShuttingDown",
					Py_BuildValue( "(d)",
						(double)shutDownTime_/Config::updateHertz() ) );

				// Don't send ack immediately to allow scripts to do stuff.
			}
			else
			{
				this->sendShutdownAck( SHUTDOWN_INFORM );
			}
		}
		break;

		case SHUTDOWN_PERFORM:
		{
			ERROR_MSG( "CellApp::controlledShutDown: "
					"CellApp does not do SHUTDOWN_PERFORM stage.\n" );
			// TODO: It could be good to call this so that we can call a script
			// method.
			break;
		}

		case SHUTDOWN_NONE:
		case SHUTDOWN_REQUEST:
		case SHUTDOWN_DISCONNECT_PROXIES:
		case SHUTDOWN_TRIGGER:
			break;
	}
}

这两个事件都会发送到Python脚本里去处理，然后在下一帧里的计时器操作中会检查是设置了shutDownTime_，如果设置了就会发送这个退出消息的ACK确认消息到对应的MgrApp里:

/**
 *	This method handles timeout events.
 */
void BaseApp::handleTimeout( TimerHandle /*handle*/, void * arg )
{
	uintptr timerType = reinterpret_cast<uintptr>( arg );

	// TODO: Should investigate whether all this can be done in tickGameTime()
	// instead, since we only seem to start a timer with TIMEOUT_GAME_TICK.

	// Secondary database is used even during shutdown
	if (pSqliteDB_ && (timerType == TIMEOUT_GAME_TICK))
	{
		pArchiver_->tickSecondaryDB( pSqliteDB_ );
	}

	bgTaskManager_.tick();

	if (this->inShutDownPause())
	{
		if (shutDownReplyID_ != Mercury::REPLY_ID_NONE)
		{
			baseAppMgr_.bundle().startReply( shutDownReplyID_ );
			baseAppMgr_.send();

			// No longer regularly sending the load from now on.
			baseAppMgr_.isRegular( false );

			shutDownReplyID_ = Mercury::REPLY_ID_NONE;
		}

		return;
	}

	switch (timerType)
	{
		case TIMEOUT_GAME_TICK:
			this->tickGameTime();
			break;
	}
}

/**
 *
 */
void CellApp::tickShutdown()
{
	if (!hasAckedCellAppMgrShutDown_)
	{
		this->sendShutdownAck( SHUTDOWN_INFORM );

		// No longer regularly sending the load from now on.
		cellAppMgr_.isRegular( false );

		hasAckedCellAppMgrShutDown_ = true;
	}
}
/**
 * 	This method sends an ack to the CellAppMgr to indicate that we've
 * 	finished one of our shutdown stages.
 */
void CellApp::sendShutdownAck( ShutDownStage stage )
{
	cellAppMgr_.ackShutdown( stage );
}

void CellAppMgrGateway::ackShutdown( ShutDownStage stage )
{
	Mercury::Bundle & bundle = channel_.bundle();
	CellAppMgrInterface::ackCellAppShutDownArgs & rAckCellAppShutDown =
		CellAppMgrInterface::ackCellAppShutDownArgs::start( bundle );

	rAckCellAppShutDown.stage = stage;

	channel_.send();
}

当对应的MgrApp收到了所有的子ServerApp的SHUTDOWN_INFORM的确认消息后，才能开启后续的SHUTDOWN_PERFORM阶段的关闭流程。

BaseAppMgr处理ACK的地方则是在SyncControlledShutDownHandler::handleMessage里，每收到一个ACK就执行一次decCount，当这个count_字段为0时，就代表所有的子BaseApp都已经ACK完毕，就会去执行ackBaseAppsShutDownToCellAppMgr来处理下一个阶段的关闭流程:

SyncControlledShutDownHandler::SyncControlledShutDownHandler(
									ShutDownStage stage, int count ) :
		stage_( stage ),
		count_( count )
{
	MF_ASSERT( count_ > 0 );
}


/**
 *	This method handles the reply message.
 */
void SyncControlledShutDownHandler::handleMessage(
		const Mercury::Address & srcAddr,
		Mercury::UnpackedMessageHeader & header,
		BinaryIStream & data, void * )
{
	this->decCount();
}

/**
 *	This method finalises and deletes this handler.
 */
void SyncControlledShutDownHandler::finalise()
{
	BaseAppMgr * pApp = BaseAppMgr::pInstance();

	if (pApp)
	{
		pApp->ackBaseAppsShutDownToCellAppMgr( stage_ );
	}

	delete this;
}


/**
 *	This method decrements the number of outstanding requests.
 */
void SyncControlledShutDownHandler::decCount()
{
	--count_;

	if (count_ == 0)
	{
		this->finalise();
	}
}

而这里的ackBaseAppsShutDownToCellAppMgr作用就是通知CellAppMgr当前所有的BaseApp都已经ACK完毕，对应的处理函数还是在之前的InformHandler里，这里会设置成员变量baseAppsAcked_为true：

/**
 *	This method is called to inform CellAppMgr that the base apps are in a
 *	particular shutdown stage.
 *	@param stage	Shutdown stage.
 */
void BaseAppMgr::ackBaseAppsShutDownToCellAppMgr( ShutDownStage stage )
{
	DEBUG_MSG( "BaseAppMgr::ackBaseAppsShutDownToCellAppMgr: "
			"All BaseApps have shut down, informing CellAppMgr\n" );
	Mercury::Bundle & bundle = this->cellAppMgr().bundle();
	CellAppMgrInterface::ackBaseAppsShutDownArgs & rAckBaseAppsShutDown =
		CellAppMgrInterface::ackBaseAppsShutDownArgs::start( bundle );

	rAckBaseAppsShutDown.stage = stage;

	this->cellAppMgr().send();
}

/**
 *	This method is called to acknowledge that the base apps are in a particular
 *	shutdown stage.
 */
void CellAppMgr::ackBaseAppsShutDown(
			const CellAppMgrInterface::ackBaseAppsShutDownArgs & args )
{
	if (pShutDownHandler_)
	{
		pShutDownHandler_->ackBaseApps( args.stage );
	}
}
void InformHandler::ackBaseApps( ShutDownStage stage )
{
	if (stage != SHUTDOWN_INFORM)
	{
		ERROR_MSG( "InformHandler::ackBaseApps: Incorrect stage %s\n", 
			ServerApp::shutDownStageToString( stage ) );
	}

	baseAppsAcked_ = true;
	this->checkStatus();
}

CellAppMgr处理ACK的地方在InformHandler::ackBaseApps，每个确认了ACK的CellApp都会放到ackedCellApps_这个集合里，当集合的大小大于等于当前管理的CellApp的大小且baseAppsAcked_为true时，就代表所有的BaseApp和所有的CellApp都接收到了SHUTDOWN_INFORM，接下来会去执行PerformBaseAppsHandler,这里会使用writeSpacesToDB将所有的Space数据进行存库，同时往BaseAppMgr发送controlledShutDown通知可以切换到SHUTDOWN_PERFORM了：

void InformHandler::checkStatus()
{
	INFO_MSG( "InformHandler::checkStatus: "
			"baseAppsAcked_ = %d. cells %" PRIzu "/%d shutDownTime %u time %u\n",
			baseAppsAcked_,
			ackedCellApps_.size(), mgr_.numCellApps(),
			shutDownTime_, mgr_.time() );

	if (this->isPaused() &&
		baseAppsAcked_ &&
		ackedCellApps_.size() >= size_t(mgr_.numCellApps()))
	{
		// TODO: Could check that the correct ones are ack'ed.
		// mgr_.setShutDownHandler( new PerformCellAppHandler( mgr_ ) );
		mgr_.setShutDownHandler( new PerformBaseAppsHandler( mgr_ ) );
		delete this;
	}
}

void InformHandler::ackCellApp( ShutDownStage stage, CellApp & app )
{
	if (stage != SHUTDOWN_INFORM)
	{
		ERROR_MSG( "InformHandler::ackCellApp: Incorrect stage %s\n",
				ServerApp::shutDownStageToString( stage ) );
	}

	ackedCellApps_.insert( &app );

	this->checkStatus();
}

PerformBaseAppsHandler::PerformBaseAppsHandler( CellAppMgr & mgr ) : mgr_( mgr )
{
	CellAppMgr & app = CellAppMgr::instance();

	app.writeGameTimeToDB();
	app.writeSpacesToDB();

	// Tell the BaseAppMgr to perform.
	{
		BaseAppMgrInterface::controlledShutDownArgs args;
		args.stage = SHUTDOWN_PERFORM;
		args.shutDownTime = 0;


		Mercury::Bundle & bundle = app.baseAppMgr().bundle();
		bundle << args;

		app.baseAppMgr().send();
	}
}

/**
 *	This method writes space information to the database.
 */
void CellAppMgr::writeSpacesToDB()
{
	if (Config::archivePeriodInTicks() == 0 ||
		!Config::shouldArchiveSpaceData())
	{
		return;
	}

	if (this->dbAppAlpha().channel().isEstablished())
	{
		Mercury::Bundle & bundle = this->dbAppAlpha().bundle();
		bundle.startMessage( DBAppInterface::writeSpaces );
		bundle << uint32( spaces_.size() );

		Spaces::const_iterator iter = spaces_.begin();

		while (iter != spaces_.end())
		{
			iter->second->sendToDB( bundle );
			++iter;
		}

		this->dbAppAlpha().send();
	}
	else
	{
		WARNING_MSG( "CellAppMgr::writeSpacesToDB: "
			"No known DBApp, not writing to DB.\n" );
	}
}

当BaseAppMgr::controlledShutDown处理SHUTDOWN_PERFORM时，会触发BaseAppMgr::startAsyncShutDownStage，这里收集当前所有的BaseApp的地址，使用AsyncControlledShutDownHandler来发送controlledShutDown通知，并等待这些BaseApp的ACK：

/**
 *	This method responds to a message telling us what stage of the controlled
 *	shutdown process the server is at.
 */
void BaseAppMgr::controlledShutDown(
		const BaseAppMgrInterface::controlledShutDownArgs & args )
{
	INFO_MSG( "BaseAppMgr::controlledShutDown: stage = %s\n", 
		ServerApp::shutDownStageToString( args.stage ) );

	switch (args.stage)
	{
		// 省略之前已经介绍过的两个分支
		case SHUTDOWN_PERFORM:
		{
			this->startAsyncShutDownStage( SHUTDOWN_DISCONNECT_PROXIES );
			break;
		}

		case SHUTDOWN_TRIGGER:
		{
			this->controlledShutDownServer();
			break;
		}

		case SHUTDOWN_NONE:
		case SHUTDOWN_DISCONNECT_PROXIES:
			break;
	}
}

void BaseAppMgr::startAsyncShutDownStage( ShutDownStage stage )
{
	BW::vector< Mercury::Address > addrs;
	addrs.reserve( baseAndServiceApps_.size() );

	BaseApps::const_iterator iter = baseAndServiceApps_.begin();

	while (iter != baseAndServiceApps_.end())
	{
		addrs.push_back( iter->first );

		++iter;
	}

	// This object deletes itself.
	new AsyncControlledShutDownHandler( stage, addrs );
}

这里的AsyncControlledShutDownHandler会遍历初始时传入的BaseApp地址集合，发送一个controlledShutDown通知，此时的stage为SHUTDOWN_DISCONNECT_PROXIES：

AsyncControlledShutDownHandler::AsyncControlledShutDownHandler(
			ShutDownStage stage, BW::vector< Mercury::Address > & addrs ) :
		stage_( stage ),
		numToSend_( 0 )
{
	addrs_.swap( addrs );
	this->sendNext();
}

void AsyncControlledShutDownHandler::sendNext()
{
	bool shouldDeleteThis = true;
	BaseAppMgr * pApp = BaseAppMgr::pInstance();

	if (pApp)
	{
		while (true)
		{
			if (numToSend_ < int(addrs_.size()))
			{
				Mercury::ChannelOwner * pChannelOwner =
					pApp->findChannelOwner( addrs_[ numToSend_ ] );

				if (pChannelOwner != NULL &&
						!pChannelOwner->channel().hasRemoteFailed())
				{
					Mercury::Bundle & bundle = pChannelOwner->bundle();
					bundle.startRequest(
							BaseAppIntInterface::controlledShutDown, this );
					shouldDeleteThis = false;
					bundle << stage_;
					bundle << 0;
					pChannelOwner->send();
				}
				else
				{
					WARNING_MSG( "AsyncControlledShutDownHandler::sendNext: "
									"Could not find channel for %s\n",
							addrs_[ numToSend_ ].c_str() );
					++numToSend_;
					continue;
				}

				++numToSend_;
			}
			else if (stage_ == SHUTDOWN_DISCONNECT_PROXIES)
			{
				// This object deletes itself.
				new AsyncControlledShutDownHandler( SHUTDOWN_PERFORM, addrs_ );
			}
			else
			{
				Mercury::Bundle & bundle = pApp->cellAppMgr().bundle();
				CellAppMgrInterface::ackBaseAppsShutDownArgs &
					rAckBaseAppsShutDown =
						CellAppMgrInterface::ackBaseAppsShutDownArgs::start(
							bundle );

				rAckBaseAppsShutDown.stage = stage_;

				pApp->cellAppMgr().send();
				pApp->shutDown( false );
			}
			break;
		}
	}

	if (shouldDeleteThis)
	{
		delete this;
	}
}

这个SHUTDOWN_DISCONNECT_PROXIES的作用是通知当前BaseApp上的所有Proxy使用onClientDeath( CLIENT_DISCONNECT_SHUTDOWN )去强制断开客户端的连接，也就是常说的关服封网:

case SHUTDOWN_DISCONNECT_PROXIES:
{
	if (this->hasStarted())
	{
		this->callShutDownCallback( 0 );

		// TODO: Should probably spread this out over time.
		typedef BW::vector< SmartPointer< Proxy > > CopiedProxies;
		CopiedProxies copyOfProxies;

		{
			copyOfProxies.reserve( proxies_.size() );

			Proxies::iterator iter = proxies_.begin();

			while (iter != proxies_.end())
			{
				copyOfProxies.push_back( iter->second );
				++iter;
			}
		}

		{
			CopiedProxies::iterator iter = copyOfProxies.begin();

			while (iter != copyOfProxies.end())
			{
				(*iter)->onClientDeath( CLIENT_DISCONNECT_SHUTDOWN );

				++iter;
			}
		}
	}


	IF_NOT_MF_ASSERT_DEV( baseAppMgr_.addr() == srcAddr )
	{
		break;
	}

	baseAppMgr_.bundle().startReply( header.replyID );
	baseAppMgr_.send();

	break;
}

当所有的Proxy都被通知断开连接后，SHUTDOWN_DISCONNECT_PROXIES阶段就完成了,开始发送消息到BaseAppMgr来通知此阶段结束，然后BaseAppMgr的AsyncControlledShutDownHandler就会切换到SHUTDOWN_PERFORM阶段:

void AsyncControlledShutDownHandler::handleMessage(
		const Mercury::Address & srcAddr,
		Mercury::UnpackedMessageHeader & header,
		BinaryIStream & data, void * )
{
	DEBUG_MSG( "AsyncControlledShutDownHandler::handleMessage: "
			"BaseApp %s has finished stage %s\n",
		srcAddr.c_str(), ServerApp::shutDownStageToString( stage_ ) );

	if (stage_ == SHUTDOWN_PERFORM)
	{
		BaseAppMgr * pApp = BaseAppMgr::pInstance();
		pApp->removeControlledShutdownBaseApp( srcAddr );
	}

	this->sendNext();
}
void AsyncControlledShutDownHandler::sendNext()
{
	bool shouldDeleteThis = true;
	BaseAppMgr * pApp = BaseAppMgr::pInstance();

	if (pApp)
	{
		while (true)
		{
			if (numToSend_ < int(addrs_.size()))
			{
				// 省略一些代码
			}
			else if (stage_ == SHUTDOWN_DISCONNECT_PROXIES)
			{
				// This object deletes itself.
				new AsyncControlledShutDownHandler( SHUTDOWN_PERFORM, addrs_ );
			}
		}
	}
}

这样又会对所有的子BaseApp发送SHUTDOWN_PERFORM阶段的消息,在对应的BaseApp::controlledShutDown会构造一个ControlledShutdown::start来开启清理流程：

void BaseApp::controlledShutDown( const Mercury::Address& srcAddr,
		const Mercury::UnpackedMessageHeader& header,
		BinaryIStream & data )
{
	ShutDownStage stage;
	GameTime shutDownTime;
	data >> stage >> shutDownTime;

	INFO_MSG( "BaseApp::controlledShutDown: stage = %s\n", 
		ServerApp::shutDownStageToString( stage ) );

	switch (stage)
	{
		// 省略其他分支
		case SHUTDOWN_PERFORM:
		{
			if (this->hasStarted())
			{
				this->callShutDownCallback( 1 );
			}

			ControlledShutdown::start( pSqliteDB_,
					bases_, localServiceFragments_,
					header.replyID, srcAddr );

			break;
		}
	}
}


namespace ControlledShutdown
{

void start( SqliteDatabase * pSecondaryDB,
			const Bases & bases,
			Bases & localServiceFragments,
			Mercury::ReplyID replyID,
			const Mercury::Address & srcAddr )
{
	localServiceFragments.discardAll();

	// This object deletes itself.
	ControlledShutDownHandler * pHandler = NULL;

	if (pSecondaryDB)
	{
		pHandler = new ShutDownHandlerWithSecondaryDB( *pSecondaryDB );
	}
	else
	{
		pHandler = new ShutDownHandlerWithoutSecondaryDB();
	}
	pHandler->init( bases, replyID, srcAddr );
}

} // namespace ControlledShutdown

这里的ShutDownHandlerWithSecondaryDB和ShutDownHandlerWithoutSecondaryDB都是ControlledShutDownHandler的子类，作用都是将所有的Base执行存库操作，它们的区别是是否有二级数据库。当所有的Base数据都存入数据库了之后，开始通知BaseAppMgr当前BaseApp的清理流程已经完成，同时当前BaseApp调用shutDown函数来执行进行最后的清理并退出:

void ControlledShutDownHandler::checkFinished()
{
	if (numOutstanding_ != 0)
	{
		return;
	}

	BaseApp * pApp = BaseApp::pInstance();

	if (pApp == NULL)
	{
		ERROR_MSG( "ControlledShutDownHandler::checkFinished: pApp is NULL\n" );
		return;
	}

	BaseAppMgrGateway & baseAppMgr = pApp->baseAppMgr();

	IF_NOT_MF_ASSERT_DEV( srcAddr_ == baseAppMgr.addr() )
	{
		return;
	}

	baseAppMgr.bundle().startReply( replyID_ );
	baseAppMgr.send();

	pApp->callShutDownCallback( 2 );

	delete this;
	pApp->shutDown();
}

当BaseAppMgr接收到了所有的BaseApp都存库完成的回应之后，开始给CellAppMgr来发送SHUTDOWN_PERFORM阶段的消息，同时这里也会调用shutDown函数来触发当前BaseAppMgr的进程退出:

void AsyncControlledShutDownHandler::sendNext()
{
	bool shouldDeleteThis = true;
	BaseAppMgr * pApp = BaseAppMgr::pInstance();

	if (pApp)
	{
		while (true)
		{
			if (numToSend_ < int(addrs_.size()))
			{
				// 省略一些代码
			}
			else if (stage_ == SHUTDOWN_DISCONNECT_PROXIES)
			{
				// This object deletes itself.
				new AsyncControlledShutDownHandler( SHUTDOWN_PERFORM, addrs_ );
			}
			else
			{
				Mercury::Bundle & bundle = pApp->cellAppMgr().bundle();
				CellAppMgrInterface::ackBaseAppsShutDownArgs &
					rAckBaseAppsShutDown =
						CellAppMgrInterface::ackBaseAppsShutDownArgs::start(
							bundle );

				rAckBaseAppsShutDown.stage = stage_;

				pApp->cellAppMgr().send();
				pApp->shutDown( false );
			}
			break;
		}
	}

	if (shouldDeleteThis)
	{
		delete this;
	}
}

当CellAppMgr接收到这个ackBaseAppsShutDown的消息时，当前的pShutDownHandler_已经是PerformBaseAppsHandler了，此时的处理就是通知DBAppMgr当前CellAppMgr的清理流程已经完成,同时调用shutDown来触发当前CellAppMgr的进程退出:

/**
 *	This method is called to acknowledge that the base apps are in a particular
 *	shutdown stage.
 */
void CellAppMgr::ackBaseAppsShutDown(
			const CellAppMgrInterface::ackBaseAppsShutDownArgs & args )
{
	if (pShutDownHandler_)
	{
		pShutDownHandler_->ackBaseApps( args.stage );
	}
}
/**
 *	This method is called when the BaseApps have all completed the perform
 *	stage.
 */
void PerformBaseAppsHandler::ackBaseApps( ShutDownStage stage )
{
	DBAppMgrInterface::controlledShutDownArgs args;
	args.stage = SHUTDOWN_PERFORM;

	mgr_.dbAppMgr().bundle() << args;

	mgr_.dbAppMgr().send();
	mgr_.setShutDownHandler( NULL );
	mgr_.shutDown( /* shutDownOthers */ true );
	delete this;
}

注意到这里并没有通知所有的CellApp去执行SHUTDOWN_PERFORM，因为CellApp压根没有SHUTDOWN_PERFORM阶段,下面这两个函数的注释里都强调了这个问题:

/**
 *	This method handles an ack from a CellApp. This should not be called because
 *	the CellApps are not involved in this stage.
 */
void PerformBaseAppsHandler::ackCellApp( ShutDownStage stage, CellApp & app )
{
	ERROR_MSG( "PerformBaseAppsHandler::ackCellApp: Got stage %s from %s\n",
		ServerApp::shutDownStageToString( stage ), app.addr().c_str() );
}

/**
 *	This method handles a message telling us to shut down in a controlled way.
 */
void CellApp::controlledShutDown(
		const CellAppInterface::controlledShutDownArgs & args )
{
	switch (args.stage)
	{
		case SHUTDOWN_INFORM:
		{
			// 省略代码
		}
		break;

		case SHUTDOWN_PERFORM:
		{
			ERROR_MSG( "CellApp::controlledShutDown: "
					"CellApp does not do SHUTDOWN_PERFORM stage.\n" );
			// TODO: It could be good to call this so that we can call a script
			// method.
			break;
		}

		case SHUTDOWN_NONE:
		case SHUTDOWN_REQUEST:
		case SHUTDOWN_DISCONNECT_PROXIES:
		case SHUTDOWN_TRIGGER:
			break;
	}
}

这些CellApp的关闭则是在PerformBaseAppsHandler::ackBaseApps里的mgr_.cellAppMgr().shutDown( true );函数调用里触发的，这个函数会遍历当前的所有CellApp，并调用每个CellApp的shutDown函数来触发发送一个shutDown请求到这个CellApp，并最终执行进程退出:

void PerformBaseAppsHandler::ackBaseApps( ShutDownStage stage )
{
	DBAppMgrInterface::controlledShutDownArgs args;
	args.stage = SHUTDOWN_PERFORM;

	mgr_.dbAppMgr().bundle() << args;

	mgr_.dbAppMgr().send();
	mgr_.setShutDownHandler( NULL );
	mgr_.shutDown( /* shutDownOthers */ true );
	delete this;
}
/**
 *	This method shuts down this application.
 */
void CellAppMgr::shutDown( bool shutDownOthers )
{
	INFO_MSG( "CellAppMgr::shutDown: shutDownOthers = %d\n", shutDownOthers );
	if (shutDownOthers)
	{
		cellApps_.shutDownAll();
	}


	INFO_MSG( "CellAppMgr::shutDown: Told to shut down. shutDownOthers = %d\n",
			shutDownOthers );
	isShuttingDown_ = true;
	this->mainDispatcher().breakProcessing();
}

void CellApps::shutDownAll()
{
	Map::iterator appIter = map_.begin();

	while (appIter != map_.end())
	{
		CellApp * pApp = appIter->second;
		pApp->shutDown();

		++appIter;
	}
}
void CellApp::shutDown()
{
	CellAppInterface::shutDownArgs args;
	args.isSigInt = false; // Not used.

	this->bundle() << args;
	this->send();
}
/**
 *	This method handles a shutDown message.
 */
void CellApp::shutDown( const CellAppInterface::shutDownArgs & /*args*/ )
{
	TRACE_MSG( "CellApp::shutDown: Told to shut down\n" );
	this->shutDown();
}

前面PerformBaseAppsHandler::ackBaseApps在CellAppMgr执行shutDown之前会通知controlledShutDown到DBAppMgr,当DBAppMgr收到这个消息的时候，就会通知所有的DbApp执行SHUTDOWN_PERFORM，并调用shutDown来触发当前DBAppMgr的进程退出:

/**
 *	This method shut down the system in a controlled manner.
 */
void DBAppMgr::controlledShutDown(
		const DBAppMgrInterface::controlledShutDownArgs & args )
{
	DEBUG_MSG( "DBAppMgr::controlledShutDown: stage = %s\n", 
		ServerApp::shutDownStageToString( args.stage ) );

	switch (args.stage)
	{
	case SHUTDOWN_REQUEST:
	{
		// 省略一些代码
	}
	
	case SHUTDOWN_PERFORM:
	{
		INFO_MSG( "DBAppMgr::controlledShutDown: "
				"Telling %" PRIzu " DBApps to shut down\n",
			dbApps_.size() );
		for (DBApps::const_iterator iter = dbApps_.begin();
				iter != dbApps_.end();
				++iter)
		{
			iter->second->controlledShutDown( SHUTDOWN_PERFORM );
		}

		this->interface().processUntilChannelsEmpty();
		this->shutDown();
		break;
	}

	default:
		ERROR_MSG( "DBAppMgr::controlledShutDown: Stage %s not handled.\n",
			ServerApp::shutDownStageToString( args.stage ) );
		break;
	}
}

现在DBAppMgr, DbApp, CellAppMgr, CellApp, BaseAppMgr, BaseApp都已经完成了有序退出，就剩LoginApp了。事实上前面的BaseAppMgr::controlledShutDownServer有通知LoginApp::controlledShutDown的代码，所以LoginApp也会收到controlledShutDown的消息，从而调用mainDispatcher_.breakProcessing触发进程退出:

void BaseAppMgr::controlledShutDownServer()
{
	if (shutDownStage_ != SHUTDOWN_NONE)
	{
		DEBUG_MSG( "BaseAppMgr::controlledShutDownServer: "
			"Already shutting down, ignoring duplicate shutdown request.\n" );
		return;
	}

	// First try to trigger controlled shutdown via the loginapp
	Mercury::Address loginAppAddr;
	Mercury::Reason reason = Mercury::MachineDaemon::findInterface(
				"LoginIntInterface", -1, loginAppAddr );

	if (reason == Mercury::REASON_SUCCESS)
	{
		Mercury::ChannelSender sender( BaseAppMgr::getChannel( loginAppAddr ) );
		Mercury::Bundle & bundle = sender.bundle();

		bundle.startMessage( LoginIntInterface::controlledShutDown );

		INFO_MSG( "BaseAppMgr::controlledShutDownServer: "
			"Triggering server shutdown via LoginApp @ %s\n",
			loginAppAddr.c_str() );

		return;
	}
	else
	{
		ERROR_MSG( "BaseAppMgr::controlledShutDownServer: "
			"Couldn't find a LoginApp to trigger server shutdown\n" );
	}
	// 省略一些代码
}

/**
 *  Handles incoming shutdown requests.  This is basically another way of
 *  triggering a controlled system shutdown instead of sending a SIGUSR1.
 */
void LoginApp::controlledShutDown( const Mercury::Address &source,
	Mercury::UnpackedMessageHeader &header,
	BinaryIStream &data )
{
	INFO_MSG( "LoginApp::controlledShutDown: "
		"Got shutdown command from %s\n", source.c_str() );

	this->controlledShutDown();
}


void LoginApp::controlledShutDown()
{
	mainDispatcher_.breakProcessing();
}

在LoginApp进程退出的时候，还会通知一下DbAppMgr执行SHUTDOWN_REQUEST:

void LoginApp::onRunComplete() /* override */
{
	INFO_MSG( "LoginApp::run: Terminating normally.\n" );
	this->ServerApp::onRunComplete();
	bool sent = false;

	if (this->isDBAppMgrReady())
	{
		Mercury::Bundle	& dbMgrBundle = dbAppMgr_.pChannelOwner()->bundle();
		DBAppMgrInterface::controlledShutDownArgs args;
		args.stage = SHUTDOWN_REQUEST;
		dbMgrBundle << args;
		dbAppMgr_.pChannelOwner()->send();
		sent = true;
	}

	if (sent)
	{
		this->intInterface().processUntilChannelsEmpty();
	}
}

但是这里DBAppMgr::controlledShutDown处理SHUTDOWN_REQUEST时，又会去通知BaseAppMgr执行SHUTDOWN_REQUEST，感觉有点没必要，因为CellApp和BaseApp发起的SHUTDOWN_REQUEST都会通知到BaseAppMgr:

void DBAppMgr::controlledShutDown(
		const DBAppMgrInterface::controlledShutDownArgs & args )
{
	DEBUG_MSG( "DBAppMgr::controlledShutDown: stage = %s\n", 
		ServerApp::shutDownStageToString( args.stage ) );

	switch (args.stage)
	{
	case SHUTDOWN_REQUEST:
	{
		// Make sure we no longer send to anonymous channels etc.
		interface_.stopPingingAnonymous();

		isShuttingDown_ = true;

		BaseAppMgrInterface::controlledShutDownArgs & args = 
			args.start( baseAppMgr_.bundle() );
		args.stage = SHUTDOWN_REQUEST;
		args.shutDownTime = 0;
		baseAppMgr_.send();

		break;
	}
	
	case SHUTDOWN_PERFORM:
	{
		// 省略一些代码
	}

	default:
		ERROR_MSG( "DBAppMgr::controlledShutDown: Stage %s not handled.\n",
			ServerApp::shutDownStageToString( args.stage ) );
		break;
	}
}

后面继续分析代码发现，LoginApp可以单独接受SIGUSR1信号，触发controlledShutDown，从而导致进程退出。此时由于不是通过CellApp或BaseApp发起的SHUTDOWN_REQUEST，所以这里LoginApp退出的时候通知DBAppMgr并间接通知BaseAppMgr执行SHUTDOWN_REQUEST是有必要的:

void LoginApp::controlledShutDown()
{
	mainDispatcher_.breakProcessing();
}
/*
 *	Override from ServerApp.
 */
void LoginApp::onSignalled( int sigNum ) /* override */
{
	this->ServerApp::onSignalled( sigNum );

	if (sigNum == SIGUSR1)
	{
		this->controlledShutDown();
	}
}

BWMachineD的清理

当一个进程对应的ServerApp执行完逻辑之后，需要从本地的BWMachineD里彻底删除这个ServerApp的注册信息。这个注册信息的删除逻辑在NetworkInterface的析构函数里。这个析构函数会通知本地的BWMachineD来执行进程的反注册并关闭所有的网络连接，最后关闭监听socket:

/**
 *	Destructor.
 */
NetworkInterface::~NetworkInterface()
{
	this->interfaceTable().deregisterWithMachined( this->address() );

	// This cancels outstanding requests. Need to make sure no more are added.
	this->finaliseRequestManager();

	pChannelMap_->destroyOwnedChannels();

	this->detach();

	this->closeSocket();

	bw_safe_delete( pDelayedChannels_ );
	bw_safe_delete( pIrregularChannels_ );
	bw_safe_delete( pKeepAliveChannels_ );
	bw_safe_delete( pCondemnedChannels_ );
	bw_safe_delete( pOnceOffSender_ );
	bw_safe_delete( pInterfaceTable_ );
	bw_safe_delete( pPacketSender_ );
	bw_safe_delete( pPacketReceiver_ );
	bw_safe_delete( pPacketLossParameters_ );
	bw_safe_delete( pDispatcher_ );
	bw_safe_delete( pChannelMap_ );
	bw_safe_delete( pRecentlyDeadChannels_ );
}

这里的deregisterWithMachined会调用之前提到的registerWithMachined方法，只是将最后一个参数isRegister设置为false，来表示这是一个反注册请求：

/**
 *	This function deregisters a socket with BWMachined.
 */
Reason deregisterWithMachined( const Address & srcAddr,
		const BW::string & name, int id )
{
	return name.empty() ?
		REASON_SUCCESS :
		registerWithMachined( srcAddr, name, id, /*isRegister:*/ false );
}

当本机的BWMachineD接收到这个反注册请求之后，会调用removeRegisteredProc方法来移除这个进程的注册信息：

case ProcessMessage::DEREGISTER:
{
	unsigned int i;
	for (i=0; i < procs_.size(); i++)
		if (pm.pid_ == procs_[i].m.pid_)
			break;

	if (i >= procs_.size())
		syslog( LOG_ERR, "Couldn't find pid %d to deregister it\n",
			pm.pid_ );
	else
		removeRegisteredProc( i );

	// confirm the deregistration to the sender
	pm.outgoing( true );
	replies.append( pm );

	syslog( LOG_INFO, "Deregistered %s for uid:%d\n",
		pm.c_str(), pm.uid_ );
	return true;
}

在这个方法里，除了会从procs数组里进行元素的移除之外，还会调用broadcastToListeners方法来广播到所有的BWMachineD进程，通知所有监听者这个进程已经死亡：

void BWMachined::removeRegisteredProc( unsigned index )
{
	if (index >= procs_.size())
	{
		syslog( LOG_ERR, "Can't remove reg proc at index %d/%" PRIzu "",
			index, procs_.size() );
		return;
	}

	ProcessInfo &pinfo = procs_[ index ];
	ProcessMessage pm;
	pm << pinfo.m;
	this->broadcastToListeners( pm, pm.NOTIFY_DEATH );

	procs_.erase( procs_.begin() + index );
}

case ProcessMessage::NOTIFY_DEATH:
{
	deathListeners_.handleNotify( this->endpoint(), pm, sin.sin_addr );
	return true;
}

走完这个流程之后，这个进程就彻底的从集群里的BWMachined里消失了，其他进程再也无法连接到这个进程了。

Mosaic Game 的玩家流程管理

玩家登录流程

在服务器完全准备好之后，就可以接受客户端的登录请求了。由于客户端只能与gate_server进行通信，所以客户端需要以某种方式去获取目标gate_server的ip:port。服务器可以用各种方式去分发当前可用的gate_server列表，这些列表里包含了每个gate_server对外服务的ip:port信息，运维人员会配置好这些ip:port的流量转发到gate_server的监听端口上。客户端选择了一个gate_server之后会将这个gate_server当作其上游进程来开启连接，连接建立之后开始创建client_session:

void basic_client::on_connect(std::shared_ptr<network::net_connection> connection)
{
	basic_stub::on_connect(connection);

	auto cur_con_name = get_connection_name(connection.get());
	if (cur_con_name && *cur_con_name == m_upstream_server.name)
	{
		m_gate_connection = connection;
		m_logger->info("set gate connection with name {}", *cur_con_name);
		if (!m_main_player)
		{
			request_create_session(connection);
		}
		else
		{
			request_reconnect_session(connection);
		}	
	}
}

这个client_session其实就是唯一会话id，同一个会话里保证网络包的顺序以及实现断线重连功能 这里会通过m_main_player是否已经初始化来判断是创建一个新的client_session还是复用之前的client_session：

void gate_server::on_request_create_session(std::shared_ptr<network::net_connection> con, std::shared_ptr<const std::string> dest, const json& msg)
{
	if(m_stopped)
	{
		return;
	}
	std::string error_info = std::string();
	std::string cur_session_str;
	std::shared_ptr<network::net_connection> outbound_con;
	do {
		if (m_connection_sessions.find(con->inbound_connection_idx) != m_connection_sessions.end())
		{
			error_info = "already has session";
			break;
		}
		outbound_con = choose_space_server();
		if (!outbound_con)
		{
			error_info = "no game server available";
			break;
		}
		cur_session_str = generate_session_str();
	} while (0);

	json reply_msg, reply_param;
	reply_msg["cmd"] = "reply_create_session";
	reply_param["errcode"] = error_info;
	if (error_info.empty())
	{
		reply_param["account_id"] = on_session_created(con, outbound_con, cur_session_str);
	}
	else
	{
		reply_param["account_id"] = std::string{};
	}
	reply_param["session"] = cur_session_str;

	reply_msg["param"] = reply_param;

	m_router->push_msg(con.get(), m_local_name_ptr, {}, std::make_shared<const std::string>(reply_msg.dump()), enums::packet_cmd_helper::encode(enums::packet_cmd::server_control, 0));


}

当客户端选择好一个gate_server后就会发送request_create_session消息，如果gate_server允许登录则会发送一个reply_create_session的消息，这个消息里会带上协商好的session与对应新创建的account_entity的account_id：

void basic_client::on_reply_create_session(std::shared_ptr<network::net_connection> con,  const json& msg)
{
	std::string errcode;
	std::string account_id;
	std::string session_str;
	try
	{
		msg.at("errcode").get_to(errcode);
		msg.at("account_id").get_to(account_id);
		msg.at("session").get_to(session_str);
	}
	catch (std::exception& e)
	{
		m_logger->error("on_reply_create_session fail to parse {} error {}", msg.dump(), e.what());
		errcode = e.what();
	}
	if (errcode.empty())
	{
		m_main_account = entity::entity_manager::instance().create_entity<entity::client_account>(account_id, 0);
		m_main_account->set_basic_client(this);
		m_main_account->init({});
		
		return;
	}
	else
	{
		m_logger->error("on_reply_create_session error {}", errcode);
		restore();
	}
}

basic_client会以这个account_id创建一个client_account，内部会创建一个登录状态机对象client_login_statem：

class  client_account final: public entity_manager::sub_class<client_account>, public utility::component_owner<account_component, client_account>
{
protected:
	std::map<utility::persist_entity_id, json> m_player_datas;
	statem::client_login_statem m_statem;
	// 省略很多代码
};

在服务端的account_entity上也有一个登录状态机login_statem:

class Meta(rpc(owner=1)) account_entity final: public entity_manager::sub_class<account_entity>, public utility::component_owner<account_component, account_entity>
{
	using account_entity_RpcSuper = utility::component_owner<account_component, account_entity>;
protected:
	std::unordered_map<utility::persist_entity_id, json> m_player_datas;
	statem::login_statem m_statem;
	// 省略很多代码
};

由于登录流程比较冗长，期间可能出现各种逻辑错误与网络错误，所以客户端与服务端都会用状态机来维护当前的登录状态。这里的登录状态机使用了一个自己做的简单实现，每个状态都继承自下面的state：

template <typename Owner, typename... Args>
class state_machine;
template <typename Owner, typename... Args>
class state
{
public:
	state_machine<Owner, Args...>& m_statem;
	dispatcher<Args...> m_dispatcher;
	virtual void on_create(){}
	virtual void on_enter(){}
	virtual void on_exit(){}

	virtual std::string name() const
	{
		return "invalid";
	}
	static std::string static_name()
	{
		return "invalid";
	}
	bool change_to(const std::string& dest_state)
	{
		return m_statem.change_to(dest_state);
	}
	template <typename K, typename T>
	void process_event(const K& event, const T& data)
	{
		m_dispatcher.dispatch(event, data);
	}
	template <typename K, typename T>
	void notify_statem(const K& event, const T& data);
public:
	state(state_machine<Owner, Args...>& in_statem)
	: m_statem(in_statem)
	{

	}
	virtual ~state()
	{

	}
	bool active() const
	{
		return m_statem.active_state() == this;
	}
};

这里的state作为所有状态的基类，提供了状态的标识接口与状态的创建与进出回调接口，子类继承的时候需要对这些virtual的接口进行复写。同时process_event接口与notify_statem接口负责提供状态与状态机之间的事件通信。以basic_client为例，登录状态机client_login_statem创建的时候会注册好各种状态，同时设置默认状态为wait_login:

client_login_statem::client_login_statem(entity::client_account* owner)
: utility::state_machine<entity::client_account, enums::account_statem_actions, std::string>(wait_login::static_name(), owner)
{
	add_state<auth_account>();
	add_state<fetch_players>();
	add_state<show_players>();
	add_state<wait_login>();
	add_state<player_online>();
	add_state<create_account>();
	add_state<replace_account>();
	add_state<logout_account>();
	change_to(wait_login::static_name());
}

客户端的输入目前通过basic_client:on_gm_cmd的input_action来通知到client_account的dispatcher:

void basic_client::on_gm_cmd(const std::string& cmd, const json& param, msg_seq_t req_seq)
{
	if (cmd == "rpc_msg")
	{
		// 省略很多代码
	}
	else if (cmd == "input_action")
	{
		std::string action_cmd;
		std::vector<json> action_args;
		try
		{
			param.at("cmd").get_to(action_cmd);
			param.at("args").get_to(action_args);
		}
		catch (std::exception& e)
		{
			auto reply = fmt::format("fail to decode action_msg {} e {}", param.dump(), e.what());
			m_logger->error("on_gm_cmd {}", reply);
			on_http_reply(req_seq, reply);
			return;
		}
		if(m_main_player)
		{
			m_main_player->set_http_reply("");
			m_main_player->dispatcher().dispatch(action_cmd, action_args);
			on_http_reply(req_seq, m_main_player->http_reply());
			return;
		}
		if (m_main_account)
		{
			m_main_account->set_http_reply("");
			m_main_account->dispatcher().dispatch(action_cmd, action_args);
			on_http_reply(req_seq, m_main_account->http_reply());
			return;
		}
		on_http_reply(req_seq, "no entity");
		
	}
	else
	{
		json_stub::on_gm_cmd(cmd, param, req_seq);

	}
}

同时client_account的所有rpc消息处理都是直接发往login_statem上，通过当前状态m_cur_state的dispatcher执行分发:

utility::rpc_msg::call_result client_account::on_rpc_msg(const utility::rpc_msg& msg) override
{
	return rpc_owner_on_rpc(msg);
}
utility::rpc_msg::call_result client_account::rpc_owner_on_rpc(const utility::rpc_msg& msg)
{
	return m_statem.on_msg(msg.cmd, msg);
}
utility::rpc_msg::call_result client_login_statem::on_msg(const std::string& cmd, const utility::rpc_msg& detail)
{
	if(!m_cur_state)
	{
		return utility::rpc_msg::call_result::rpc_not_found;
	}
	if(m_cur_state->m_dispatcher.dispatch(cmd, detail))
	{
		return utility::rpc_msg::call_result::suc;
	}
	return utility::rpc_msg::call_result::rpc_not_found;
}

每个状态创建的时候都会注册自己感兴趣的相关事件回调，这里既可以注册到状态机自身的state::m_dispatcher上也可以注册到owner()->dispatcher()上，m_dispatcher上的事件对应rpc处理，owner()->dispatcher()对应输入处理：

void wait_login::on_create()
{

	owner()->dispatcher().add_listener(std::string("auth_account"), make_active_listener(&wait_login::try_auth_account, this));
	owner()->dispatcher().add_listener(std::string("create_account"), make_active_listener(&wait_login::try_create_account, this));

}
void replace_account::on_create()
{
	m_dispatcher.add_listener<std::string, utility::rpc_msg, replace_account>("reply_replace_account", &replace_account::on_reply_replace_account, this);
	owner()->dispatcher().add_listener(std::string("replace_account"), make_active_listener(&replace_account::try_replace_account, this));
}

但是处理owner()->dispatcher()的时候需要判断自己这个state的状态是不是active的，这样才能维护好状态之间的跳转，所以这里使用了一个封装函数make_active_listener，这样就避免每次在真正的回调里去判断是否是active的:

template<typename K, typename V, typename S>
static std::function<void(const K&, const V&)>  make_active_listener(void(S::*cur_callback)(const K&, const V&), S* self)
{
	return [=](const K& event, const V& arg)
	{
		if(!self->active())
		{
			self->owner()->logger()->error("login state {} not active when handle cmd {}", self->name(), event);
			return;
		}
		else
		{
			(self->*cur_callback)(event, arg);
		}
	};
}

账号的登录状态机初始状态为wait_login，在初始的空白账号情况下，wait_login需要先通过create_account来创建一个账号，当创建账号成功的时候会自动切换到展示空角色列表的状态show_players中：

void wait_login::try_create_account(const std::string& cmd, const std::vector<json>& data)
{
	if(!active())
	{
		owner()->logger()->error("login state {} not active when handle cmd {}", name(), cmd);
		return;
	}
	auto cur_owner = owner();
	

	std::string account_name;
	std::string account_passwd;
	try
	{
		data.at(0).get_to(account_name);
		data.at(1).get_to(account_passwd);
	}
	catch (const std::exception& e)
	{
		cur_owner->logger()->error("try_create_account fail to parse {} error {}", json(data).dump(), e.what());
		return;
	}
	cur_owner->try_create_account(account_name, account_passwd);
	if(!change_to("create_account"))
	{
		owner()->logger()->info("wait_login change to create_account false");	
		return;
	}
	owner()->logger()->info("wait_login change to create_account true");
}

void create_account::on_reply_create_account(const std::string& cmd, const utility::rpc_msg& data)
{
	if(!active())
	{
		owner()->logger()->error("login state {} not active when handle cmd {}", name(), cmd);
		return;
	}
	if (!data.err.empty())
	{
		change_to("wait_login");
		return;
	}
	
	else
	{
		owner()->logger()->info("on_reply_create_account enter show_players");
		change_to("show_players");
	}
}

cur_owner->try_create_account会发起一个rpc，并最终转移到服务端登录状态机里的wait_login状态去处理：

void wait_login::try_create_account(const std::string& cmd, const utility::rpc_msg& data)
{
	owner()->logger()->info("try_create_account with data {}", json(data).dump());
	auto cur_owner = owner();
	std::uint32_t cmd_err = 0;

	std::string account_name;
	std::string account_passwd;
	try
	{
		data.args.at(0).get_to(account_name);
		data.args.at(1).get_to(account_passwd);
	}
	catch (const std::exception& e)
	{
		cur_owner->logger()->error("try_create_account fail to parse {} error {}", json(data).dump(), e.what());
		cmd_err = int(account_errcodes::invalid_msg);
	}
	
	if (cmd_err == int(account_errcodes::ok))
	{
		utility::rpc_msg cur_msg;
		cur_msg.cmd = "request_create_account";
		cur_msg.args.push_back(account_name);
		cur_msg.args.push_back(account_passwd);
		owner()->call_service("login_service", cur_msg);
		change_to("create_account");
	}
	else
	{
		utility::rpc_msg return_info;
		return_info.cmd = "reply_create_account";
		return_info.err = std::to_string(cmd_err);
		return_info.args.push_back(account_name);
		owner()->call_client(return_info);
	}
}

如果参数校验通过，则会将账号密码信息发送到登录服务login_service上去执行。如果login_service上的创建账号检查通过，会返回一个reply_create_account的rpc到服务端的account_entity，此时服务端的登录状态机的create_account状态会接收这个rpc来执行状态切换以及消息下发：

void create_account::on_create()
{
	m_dispatcher.add_listener<std::string, utility::rpc_msg>("reply_create_account", &create_account::reply_create_account, this);

}


void create_account::reply_create_account(const std::string& cmd, const utility::rpc_msg& detail)
{
	std::string account_name;
	utility::rpc_msg return_info;
	return_info.err = detail.err;
	try
	{
		detail.args.at(0).get_to(account_name);
	}
	catch(std::exception& e)
	{
		owner()->logger()->error("on_create_account_back fail to parse  with error {}", e.what());
		return_info.err = "invalid server msg";
	}
	
	
	return_info.cmd = "reply_create_account";
	owner()->call_client(return_info);
	if(return_info.err.empty())
	{
		owner()->set_account(account_name);
		owner()->dispatcher().dispatch(enums::event_category::account, std::string("auth_account_suc"));
		change_to("show_players");
	}
	else
	{
		owner()->dispatcher().dispatch(enums::event_category::account,std::string("auth_account_fail"));
		change_to("wait_login");
	}
}

这个reply_create_account的rpc传递到客户端之后，会由客户端登录状态机的create_account状态来接力处理，这样就是客户端与服务端的登录状态机的典型交互流程：

void create_account::on_create()
{
	m_dispatcher.add_listener<std::string, utility::rpc_msg, create_account>("reply_create_account", &create_account::on_reply_create_account, this);
}



void create_account::on_reply_create_account(const std::string& cmd, const utility::rpc_msg& data)
{
	if (!data.err.empty())
	{
		change_to("wait_login");
		return;
	}
	
	else
	{
		owner()->logger()->info("on_reply_create_account enter show_players");
		change_to("show_players");
	}
}

注意这里的password是明文密码，为了增强账号密码的安全性最好对密码执行一个单向变换生成一个password_hash字符串，最简单的就是password_hash=md5(password)，或者password_hash=md5(account_name+password)。更优的策略是每次账号注册的时候给这个account生成一个随机的长字符串salt，同时存储到数据库之中。在账号创建时，对密码做加盐变换password_hash=md5(password+salt)，将(account,password_hash,salt)这个三元组同时存储在数据库中，这样就可以有效避免数据库泄露之后的暴力破解。如果网络连接是未加密状态的话，客户端需要提前做一次password=md5(password)这样的映射，避免网络中出现明文密码。

如果登录一个已经创建的账号，则会通过try_auth_account来执行账号验证，如果账号验证通过则会切换到获取玩家列表的状态fetch_players：

void wait_login::try_auth_account(const std::string& cmd, const std::vector<json>& data)
{
	if(!active())
	{
		owner()->logger()->error("login state {} not active when handle cmd {}", name(), cmd);
		return;
	}
	auto cur_owner = owner();

	std::string account_name;
	std::string account_passwd;
	try
	{
		data.at(0).get_to(account_name);
		data.at(1).get_to(account_passwd);
	}
	catch (std::exception& e)
	{
		cur_owner->logger()->error("try_auth_account fail to parse {} error {}", json(data).dump(), e.what());
		return;
	}
	cur_owner->try_auth_account(account_name, account_passwd);
	change_to("auth_account");

}

void auth_account::on_reply_auth_account(const std::string& cmd, const utility::rpc_msg& data)
{
	if(!active())
	{
		owner()->logger()->error("login state {} not active when handle cmd {}", name(), cmd);
		return;
	}
	
	if(data.err.empty())
	{
		owner()->enum_dispatcher().dispatch(enums::event_category::account, std::string("auth_account_suc"));
		change_to("fetch_players");
	}
	else
	{
		if(data.err == "replace account needed")
		{
			change_to("replace_account");
		}
		else
		{
			owner()->enum_dispatcher().dispatch(enums::event_category::account, std::string("auth_account_fail"));
			change_to("wait_login");
		}
		
	}
}

在fetch_players状态下，服务端会去拉取这个账号下创建的所有角色列表，当角色列表数据返回之后，就会切换到展示角色列表的状态show_players:

void fetch_players::on_reply_fetch_players(const std::string& cmd, const utility::rpc_msg& player_data)
{
	if(!active())
	{
		owner()->logger()->error("login state {} not active when handle cmd {}", name(), cmd);
		return;
	}
	
	if (!player_data.err.empty())
	{
		owner()->enum_dispatcher().dispatch(enums::event_category::account, std::string("fetch_players_fail"));
		return;
	}
	std::vector<json::object_t> temp_player_data;
	try
	{
		player_data.args.at(0).get_to(temp_player_data);
	}
	catch (std::exception& e)
	{
		owner()->logger()->error("on_fetch_players_back fail to parse args: {} error: {}", json(player_data.args).dump(), e.what());
		return;
	}
	owner()->set_player_datas(temp_player_data);
	m_statem.change_to("show_players");
	owner()->enum_dispatcher().dispatch(enums::event_category::account, std::string("fetch_players_suc"));
	
}

在show_players状态下，支持对角色的增删，以及选择一个角色进行become_player上线操作：

void show_players::on_create()
{
	owner()->dispatcher().add_listener(std::string("create_player"),make_active_listener(&show_players::try_create_player, this));
	owner()->dispatcher().add_listener(std::string("delete_player"), make_active_listener(&show_players::try_delete_player, this));
	owner()->dispatcher().add_listener(std::string("become_player"),make_active_listener(&show_players::try_become_player, this));
	owner()->dispatcher().add_listener(std::string("logout_account"),make_active_listener(&show_players::try_logout_account, this));
	owner()->dispatcher().add_listener(std::string("show_players"),make_active_listener(&show_players::on_show_players, this));

	m_dispatcher.add_listener<std::string, utility::rpc_msg, show_players>("reply_create_player", &show_players::on_reply_create_player, this);
	m_dispatcher.add_listener<std::string, utility::rpc_msg, show_players>("reply_delete_player", &show_players::on_reply_delete_player, this);
	m_dispatcher.add_listener<std::string, utility::rpc_msg, show_players>("reply_become_player", &show_players::on_reply_become_player, this);
}

这个become_player成功之后，服务端与客户端都会以这个player的数据执行player_entity的创建，同时切换到player_online状态：

void show_players::on_reply_become_player(const std::string& cmd, const utility::rpc_msg& detail)
{
	m_pending_cmd.clear();
	if(!detail.err.empty())
	{
		owner()->enum_dispatcher().dispatch(enums::event_category::account,std::string("become_player_fail"));
		return;
	}
	utility::persist_entity_id player_id;
	json::object_t player_init_info;
	try
	{
		detail.args.at(0).get_to(player_id);
		detail.args.at(1).get_to(player_init_info);

	}
	catch (const std::exception& e)
	{
		owner()->logger()->warn("on_reply_become_player decode fail {} error {}", json(detail.args).dump(), e.what());

		return;
	}
	m_statem.change_to("player_online");
	owner()->on_become_player(player_id, player_init_info);
}

玩家登出流程

在player_online状态，唯一支持的操作是退出当前玩家的登录：

void player_online::try_logout_player(const std::string& cmd, const std::vector<json>& data)
{
	owner()->logger()->info("player_online::try_logout_player");
	owner()->try_logout_player();
	owner()->enum_dispatcher().dispatch(enums::event_category::account, std::string("logout"));

}

void client_account::try_logout_player()
{
	return call_server("request_logout_player", {});
}

在client_account::try_logout_player函数中，只会发送一个request_logout_player的RPC到服务端的account_entity去处理。这里会检查当前是否已经创建了player_entity，如果已经有player_entity，则需要先执行这个player_entity的注销：

void account_entity::request_logout_account(const utility::rpc_msg& msg)
{
	if(m_statem.active_state_name() == "logout_account")
	{
		return;
	}
	if(m_player_id.empty())
	{
		utility::rpc_msg request_msg;
		request_msg.cmd = "request_logout_account";
		call_service("login_service", request_msg);
		dispatcher().dispatch(enums::event_category::account, "logout");
		m_statem.change_to("logout_account");
	}
	else
	{
		utility::rpc_msg request_msg;
		request_msg.cmd = "request_logout_player";
		call_player(request_msg);
		m_statem.change_to("logout_account");
	}
	
}

不管有没有对应的在线玩家，这个RPC执行之后都会强行切换到logout_account状态下,此时会等待退出登录的RPC返回，然后切换到未登录wait_login状态。

void logout_account::reply_logout_account(const std::string& cmd, const utility::rpc_msg& data)
{
	owner()->call_client(data);
	owner()->finish_logout();
	m_statem.change_to("wait_login");
}

上面的流程对应的是客户端主动发起的退出登录操作，实际上游戏内会有服务端主动将一个玩家踢出登录的需求，因此在服务端登录状态机player_online监听了退出登录的RPC:

void player_online::on_create()
{
	m_dispatcher.add_listener<std::string, utility::rpc_msg, player_online>("notify_player_logout", &player_online::notify_player_logout, this);
}


void player_online::on_enter()
{
	
}
void player_online::notify_player_logout(const std::string& cmd, const utility::rpc_msg& data)
{
	change_to("show_player");
	owner()->dispatcher().dispatch(enums::event_category::account, std::string("logout_player"));
	owner()->call_client(data);
}

但是此时只是在客户端与服务端同时删除了对应的在线玩家角色player_entity，账号account_entity是依然在线的，登录状态机会切换到show_player状态，即账号仍然在线。如果要彻底回到登录前状态，需要在show_players状态下发起logout_account请求，这样才会将服务端客户端的account_entity彻底销毁，同时服务端客户端之间的网络连接也会断开。

void show_players::try_logout_account(const std::string& cmd, const std::vector<json>& data)
{	
	owner()->try_logout_account();
	m_statem.change_to("logout_account");
}

玩家进出场景与迁移流程

当客户端登录完成并创建出对应的player_entity之后，还需要进入游戏场景中与环境和其他玩家进行交互。这个进入场景的操作由player_space_component::request_enter_space这个rpc来发起：

void player_space_component::request_enter_space(const utility::rpc_msg& msg, std::uint32_t space_no, const std::string& space_id, json::object_t& enter_info);

这里rpc制定了要进入的场景编号space_no和场景实例space_id，enter_info里可以携带进入场景的位置与朝向信息。在各项检查都通过之后，player_entity会向管理场景的服务space_service发出进入场景请求，同时将正在进入的场景编号记录在当前玩家的属性上，避免rpc返回之前发起下一个切换场景的请求：

if(cur_space)
{
	leave_space_impl();
}
utility::rpc_msg request_msg;
request_msg.cmd = "request_enter_space";

request_msg.args.push_back(m_owner->entity_id());
request_msg.args.push_back(m_player->prop_data().team().id());
request_msg.args.push_back(space_no);
request_msg.args.push_back(space_id);
request_msg.args.push_back(enter_info);
m_owner->call_service("space_service", request_msg);
m_player->prop_proxy().space().entering_space_no().set(space_no);
return;

如果玩家当前已经在一个场景中，此时还会自动的触发场景的离开操作：

void player_space_component::leave_space_impl()
{
	utility::rpc_msg notify_service_msg;
	notify_service_msg.cmd = "report_leave_space";
	notify_service_msg.args.push_back(m_owner->entity_id());
	m_owner->call_service("space_service", notify_service_msg);
	auto cur_space = m_owner->get_space();
	cur_space->leave_space(m_owner);
}

这里的space_service::report_leave_space里会在space_service上解除当前玩家绑定的场景，而request_enter_space则会添加玩家与指定场景的映射关系，因此需要先执行leave操作，才能去执行enter。

值得注意的是这里的space_id可以传递空字符串，代表选择当前space_no的任一场景实例进入即可，对应的常见的进入主城场景的任意分线。如果space_id不为空且对应的场景实例不存在，这个请求将会执行失败。如果对应的场景实例存在，且各种场景进入的检查都通过了，space_service会发送reply_enter_space到这个player_entity上：

if(!msg.err.empty())
{
	m_player->prop_proxy().space().entering_space_no().set(0);
	return;
}

if(game_id == m_owner->get_local_server_name())
{

	auto new_space = server::space_manager::instance().get_space(space_id);
	if(!new_space)
	{
		m_owner->logger()->error("cant find space {}", space_id);
		return;
	}
	new_space->enter_space(m_owner, enter_info);
	return;
}
else
{
	utility::rpc_msg request_msg;
	request_msg.cmd = "request_migrate_begin";
	request_msg.args.push_back(game_id);
	request_msg.args.push_back(space_id);
	request_msg.args.push_back(union_space_id);
	request_msg.args.push_back(enter_info);
	m_owner->call_relay_anchor(request_msg);
}

这里首先把之前记录的正在进入的场景编号清空，然后判断要进入的场景是否是同一个进程上。如果是同一个进程，则直接调用这个space_entity::enter_space即可，这个函数里会执行一些数据的记录以及玩家的位置设置，并触发AOI的重新计算。

如果不是同一个进程，则首先需要将当前玩家迁移到目标进程，此时会将玩家的所有数据进行打包，但是打包之前需要先用request_migrate_begin通知对应的relay_anchor设置当前player_entity的状态为迁移中，避免迁移时的消息丢失。至于relay_anchor的具体作用将在后面进行详细解释。

当relay_anchor设置后迁移中的状态之后，会发送reply_migrate_begin回当前的player_entity，此时会通知当前进程的space_manager来执行迁移出去的操作：

void player_space_component::reply_migrate_begin(const utility::rpc_msg& msg, const std::string& game_id, const std::string& space_id, const std::string& union_space_id, const json::object_t& enter_info)
{
	auto new_enter_info = enter_info;
	server::space_manager::instance().migrate_out(m_owner, game_id, space_id, union_space_id, new_enter_info);
}

void space_manager::migrate_out(entity::actor_entity *cur_entity, const std::string &game_id, const std::string &space_id, const std::string &union_space_id, json::object_t &enter_info)
{
	json::object_t migrate_info;
	bool enter_new_space = true;
	auto pre_space = cur_entity->get_space();
	if(pre_space && pre_space->union_space_id() == union_space_id)
	{
		enter_new_space = false;
		enter_info["pos"] = cur_entity->pos();
		enter_info["yaw"] = cur_entity->yaw();
	}
	if(pre_space && enter_new_space)
	{
		pre_space->leave_space(cur_entity);
	}
	enter_info["enter_new_space"] = enter_new_space;
	cur_entity->migrate_out(migrate_info, enter_new_space);
	utility::rpc_msg cur_msg;
	cur_msg.cmd = "migrate_in";
	m_logger->info("migrate out entity {} to game {} space {} union_space_id {} with info {} enter_new_space {} ", cur_entity->entity_id(), game_id, space_id, union_space_id, json(migrate_info).dump(), enter_new_space);
	cur_msg.args.push_back(cur_entity->entity_id());
	cur_msg.args.push_back(cur_entity->online_entity_id());
	cur_msg.args.push_back(cur_entity->m_base_desc.m_type_name);
	cur_msg.args.push_back(space_id);
	cur_msg.args.push_back(union_space_id);
	cur_msg.args.push_back(enter_info);
	cur_msg.args.push_back(migrate_info);
	if(enter_new_space)
	{
		m_server->destroy_entity(cur_entity);
	}
	
	m_server->call_server(utility::rpc_anchor::concat(game_id,  "space_manager"), cur_msg);
}

这个space_manager::migrate_out会通知当前player_entity先退出已有场景，然后通过player_entity::migrate_out接口来将所有需要迁移的数据打包到migrate_info中，数据打包完成之后会往目标进程的space_manager发送一个migrate_in请求。

void actor_entity::migrate_out(json::object_t& migrate_info, bool enter_new_space)
{
	encode_migrate_out_data(migrate_info, enter_new_space);
	
	
	auto migrate_out_lambda = [=](actor_component* cur_comp)
	{
		cur_comp->migrate_out(enter_new_space);
	};
	call_component_interface(migrate_out_lambda);
	if(!enter_new_space)
	{
		m_is_ghost = true;
		auto become_ghost_lambda = [=](actor_component* cur_comp)
		{
			cur_comp->on_become_ghost();
		};
		call_component_interface(become_ghost_lambda);
	}
	
}

发送到目标进程之后再进行解包并重新创建player_entity，然后执行player_entity::migrate_in操作来恢复一些打包的数据：

if(enter_new_space)
{
	cur_entity = dynamic_cast<entity::actor_entity*>(m_server->create_entity(type_id, entity_id, online_entity_id, init_info, error));
	if (!cur_entity)
	{
		m_logger->error("fail to create entity {} type {} with error {}", entity_id, type_id, error);
		return;
	}
}
else
{
	// 省略一些无缝大世界代码
}

cur_entity->migrate_in(init_info, enter_new_space);
if(enter_new_space)
{
	cur_space_iter->second->enter_space(cur_entity, enter_info);
}

最后利用打包的场景数据enter_info来执行玩家的进入场景操作，至此一个带迁移的场景切换流程就结束了。

玩家断线流程

玩家的网络状态是不可控的，特别是使用移动网络的手机设备在执行位置移动时可能会出现基站的切换从而导致连接的断开。如果粗暴的将连接断开处理为客户端被动下线的话，玩家的体验就会变得非常的差。因为重新走一遍登录流程来继续游戏会导致服务端角色的销毁后再重新创建，会浪费好几秒的时间，同时玩家状态的重新同步也是非常耗流量的。为了提升玩家在网络突然断开然后又重新连接时的游戏体验，减少重新全同步时的流量成本，现在的游戏服务器基本都会有断线重连这个功能。断线重连功能需要服务端和客户端一起协作：

1. 服务端在往客户端发消息的时候，每个包都赋予一个唯一递增序列号，同时本地用滑动窗口的形式存储最近一小段时间内发送到客户端的所有消息，
2. 当服务端发现客户端断线时，需要将下发到这个客户端的消息进行缓存住，避免消息的丢失，
3. 客户端在发现自己断线之后，立即开启连接到服务器的重试，重试的时候带上最后收到的服务端消息包序列号
4. 服务端收到重连的消息请求时，根据上发过来的包序列号进行比对，
   1. 如果这个包序列号在滑动窗口内，则重连成功，服务端将这个序列号之后的所有消息执行重发
   2. 如果这个包序列号不在滑动窗口内，则重连失败，服务端通知客户端重新走登录流程

上面介绍的只是一个大概的断线重连步骤，实际的实现中会涉及到很多的细节，这些细节将在网络相关章节中具体阐述。

玩家顶号流程

当一个账号A已经在某个客户端B上成功执行了登录后，玩家选择在另外一个客户端C上尝试登录同样的账号A时，会收到该账号已经在线的错误提示。如果我们要求账号必须要在不在线的时候执行登录，则玩家需要手动在设备B上执行登出操作，然后再在客户端C上执行登录操作，这个流程会伴随着服务端角色的销毁与重建，以及新客户端的重新同步，也是非常影响玩家的体验以及浪费服务端资源。面对这种情景，最好的方式是如果客户端C收到了账号已经在线的错误提示，可以选择执行顶号操作：即通知服务端将这个角色的客户端连接由B切换为C，这样就可以避免服务端角色的销毁与创建。因此在auth_account状态下收到账号已经在线的通知时，会自动切换到replace_account状态：

void auth_account::on_reply_auth_account(const std::string& cmd, const utility::rpc_msg& data)
{
	
	if(data.err.empty())
	{
		owner()->enum_dispatcher().dispatch(enums::event_category::account, std::string("auth_account_suc"));
		change_to("fetch_players");
	}
	else
	{
		if(data.err == "replace account needed")
		{
			change_to("replace_account");
		}
		else
		{
			owner()->enum_dispatcher().dispatch(enums::event_category::account, std::string("auth_account_fail"));
			change_to("wait_login");
		}
		
	}
}

在replace_account状态下接收到客户端发起的顶号确认时，会发送顶号try_replace_account这个RPC到服务器，这里需要带上被顶号的账号other_account_id,这个参数在上面的reply_auth_account会下发：

void replace_account::try_replace_account(const std::string& cmd, const std::vector<json>& data)
{
	std::string account_name;
	std::string other_account_id;
	try
	{
		data.at(0).get_to(account_name);
		data.at(1).get_to(other_account_id);
	}
	catch(const std::exception& e)
	{
		owner()->logger()->error("try_replace_account fail to parse {} with error {}", json(data).dump(), e.what());
		return;
	}
	owner()->try_replace_account(account_name, other_account_id);
}

服务端对应的状态机replace_account会将这个请求转发到管理登录与在线的服务login_service上，检查通过之后会通知到replace_account状态机开启顶号流程：

void login_service::replace_account_impl(const std::string& new_account_id, const std::string& account_name, const std::string& gate_name)
{
	auto pre_account_info = m_online_accounts[account_name];
	utility::rpc_msg reply;
	reply.cmd = "notify_account_client_replaced";
	reply.args.push_back(new_account_id);
	reply.args.push_back(gate_name);
	
	auto cur_server = get_server();
	cur_server->call_server(this, pre_account_info.account_id, reply);
}
void account_entity::notify_account_client_replaced(const utility::rpc_msg& msg, const std::string& new_account, const std::string& new_gate_name)
{
	m_logger->info("{} notify_account_client_replaced new_account {} new_gate {}", m_base_desc.m_persist_entity_id, new_account, new_gate_name);
	get_server()->rebind_gate_client_begin(this, new_account, new_gate_name);
}

顶号的时候需要通知原有的pre_account_info.account_id对应的account_entity重新绑定到新的客户端连接上，这个客户端连接的标识符就是(new_gate_name,new_account)这两个字符串组成的pair:

void space_server::rebind_gate_client_begin(entity::account_entity* cur_account, const std::string& new_account, const std::string& new_gate_name)
{
	auto temp_iter = new_account.rfind(utility::rpc_anchor::seperator);
	if(temp_iter == std::string::npos)
	{
		m_logger->error("invalid new_account id {}", new_account);
		return;
	}
	auto new_account_id = new_account.substr(temp_iter + 2);
	// 这里可能出现同进程的情况
	auto new_account_gate_con = m_router->get_connection_with_name(new_gate_name);
	if(!new_account_gate_con)
	{
		m_logger->error("fail to find gate connection for client {}", new_gate_name);
		return;
	}
	json replace_msg, replace_param;

	// 通知老的client 对应的gate 被顶号
	auto old_gate_id = cur_account->get_gate_id();
	auto new_gate_id = get_connection_inbound_idx(new_account_gate_con);
	replace_msg["cmd"] = "notify_client_remove_account";
	replace_msg["param"] = json::object_t();
	m_router->push_msg(old_gate_id, m_local_name_ptr, cur_account->get_shared_global_id(), std::make_shared<std::string>(replace_msg.dump(4)), enums::packet_cmd_helper::encode(enums::packet_cmd::server_control, 0));

	// 通知新的account对应的entity顶号成功 触发这个account的自动销毁
	utility::rpc_msg replace_info;
	replace_info.cmd = "reply_replace_account";
	replace_info.args.push_back(cur_account->is_player_online());
	call_server(cur_account, new_account, replace_info);

	replace_msg["cmd"] = "notify_client_rebind_account";
	replace_param["pre_connected_account"] = new_account_id;
	replace_param["new_connected_account"] = cur_account->entity_id();
	replace_msg["param"] = replace_param;
	m_router->push_msg(new_gate_id, m_local_name_ptr, {}, std::make_shared<std::string>(replace_msg.dump(4)), enums::packet_cmd_helper::encode(enums::packet_cmd::server_control, 0));
	// 切换entity的gate 开启客户端掉线的timer 等待重新绑定之后会取消这个timer
	m_gate_entities[old_gate_id].erase(cur_account);
	cur_account->set_gate(std::string{}, 0);

}

在这个执行顶号的核心函数中，主要完成了如下任务：

1. 使用notify_client_remove_account，通知老的client连接，其连接因为顶号被断开，避免其触发断线重连

void gate_server::on_notify_client_remove_account(std::shared_ptr<network::net_connection> con, std::shared_ptr<const std::string> dest, const json& msg)
{
	if(!dest)
	{
		return;
	}
	json replay_msg;
	replay_msg["msg"] = "client replaced";
	request_client_close_impl(*dest, replay_msg);
}

2. 使用reply_replace_account通知新的account其顶号成功，并触发新account的自动销毁

void replace_account::on_reply_replace_account(const std::string& cmd, const utility::rpc_msg& data)
{
	if(!data.err.empty())
	{
		owner()->call_client(data);
		change_to("wait_login");
		return;
	}
	else
	{
		// 顶号成功时 需要在切换完client之后再下发这个顶号成功消息
		owner()->on_replace_account_suc(data);
	}
}

void account_entity::on_replace_account_suc(const utility::rpc_msg& msg)
{
	m_logger->info("{} on_replace_account_suc {} ", m_base_desc.m_persist_entity_id, json(msg.args).dump());
	get_server()->replace_client_remove_account(this);
}
void space_server::replace_client_remove_account(entity::account_entity* cur_account)
{
	auto cur_gate_id = cur_account->get_gate_id();
	m_gate_entities[cur_gate_id].erase(cur_account);
	entity::entity_manager::instance().destroy_entity(cur_account);
}

3. 使用notify_client_rebind_account通知新client对应的gate将其绑定的account的地址切换为老的account地址，这个rpc函数如果执行成功，会以notify_rebind_gate_client_finish这个rpc来通知回当前的account_entity

4. 通过set_gate来清空当前account的gate信息，触发创建自动倒计时销毁的计时器，同时通知在线玩家客户端已经丢失

void account_entity::set_gate(const std::string& gate_name, std::uint64_t gate_id, bool during_replace)
{
	m_gate_name = gate_name;
	m_gate_id = gate_id;
	m_relay_entity->setup_client_info(m_gate_id, get_call_proxy());
	if(gate_id != 0)
	{
		// 设置新的有效gate部分代码 此处先暂时省略
	}
	else
	{
		if(is_player_online())
		{
			utility::rpc_msg client_destory_msg;
			client_destory_msg.cmd = "notify_player_client_destroyed";
			call_player(client_destory_msg);
		}
		cancel_timer(m_destroy_client_timer);
		m_destroy_client_timer.reset();
		m_destroy_client_timer = add_timer_with_gap(std::chrono::seconds(m_auto_logout_second_when_client_destroy), [this]()
		{
			request_logout_account(utility::rpc_msg());
		});
	}
}

当notify_rebind_gate_client_finish通知回当前的account_entity之后，完整的顶号流程就算完成了，这里会通过set_gate重更新绑定一下新client连接对应的gate，并删除之前添加的自动销毁计时器，同时通知对应的在线玩家连接到了新的客户端：

void space_server::notify_rebind_gate_client_finish(std::shared_ptr<network::net_connection> con, std::shared_ptr<const std::string> from, const json& msg)
{
	std::string entity_id;
	try
	{
		msg.at("entity_id").get_to(entity_id);
	}
	catch (std::exception& e)
	{
		m_logger->error("notify_rebind_gate_client_finish fail parse {} error {}", msg.dump(), e.what());
		return;
	}
	auto cur_account = entity::entity_manager::instance().get_entity<entity::account_entity>(entity_id);
	if (!cur_account)
	{
		m_logger->error("notify_rebind_gate_client_finish cant find account entity {}", entity_id);
		return;
	}
	if(cur_account->get_gate_id() != 0)
	{
		m_logger->error("notify_rebind_gate_client_finish {} expect gate_id 0 while meet", entity_id, cur_account->get_gate_id());
		return;
	}
	auto cur_con_id = get_connection_inbound_idx(con.get());
	m_gate_entities[cur_con_id].insert(cur_account);

	// 触发重新同步数据
	cur_account->set_gate(*get_connection_name(con.get()), cur_con_id, true);		
}

void account_entity::set_gate(const std::string& gate_name, std::uint64_t gate_id, bool during_replace)
{
	m_gate_name = gate_name;
	m_gate_id = gate_id;
	m_relay_entity->setup_client_info(m_gate_id, get_call_proxy());
	if(gate_id != 0)
	{
		m_logger->info("{} notify_rebind_gate_client_finish with new_gate {}", m_base_desc.m_persist_entity_id, gate_name);
		cancel_timer(m_destroy_client_timer);
		m_destroy_client_timer.reset();
		if(during_replace)
		{
			utility::rpc_msg replace_info;
			replace_info.cmd = "reply_replace_account";
			replace_info.args.push_back(is_player_online());
			call_client(replace_info);
		}
		if(!is_player_online())
		{
			m_statem.change_to("show_players");
		}
		else
		{
			// 触发重新同步数据
			utility::rpc_msg account_replace_msg;
			account_replace_msg.cmd = "notify_player_client_replaced";
			account_replace_msg.args.push_back(m_gate_name);
			account_replace_msg.args.push_back(m_relay_entity->gate_version());
			call_player(account_replace_msg);
		}
	}
	// 省略之前介绍的客户端断线部分
}

这里的set_gate传递了during_replace=true，因此会向client发送顶号彻底完成的消息，让其客户端登录状态机进行状态切换。

当服务端玩家player_entity收到其连接到新客户端的notify_player_client_replaced请求之后，会将当前玩家的数据打包之后进行重新同步：

void player_entity::notify_player_client_replaced(const utility::rpc_msg& msg, const std::string& new_gate_name, std::uint8_t new_gate_version)
{
	m_logger->warn("player client replaced by new gate {} version {}", new_gate_name, new_gate_version);
	m_gate_version = new_gate_version;
	if(new_gate_name.empty())
	{
		return;
	}

	auto sync_info = encode_with_flag(std::uint32_t(enums::encode_flags::self_client));
	utility::rpc_msg full_sync_msg;
	full_sync_msg.cmd = "create_player";
	full_sync_msg.args.push_back(entity_id());
	full_sync_msg.args.push_back(std::move(sync_info));
	call_client(full_sync_msg);
	m_login_dispatcher.dispatch(true);

}

当客户端接收到这个create_player数据之后，就会创建出对应的玩家、所在的场景以及周围的其他同步entity。

BigWorld 的玩家流程管理

客户端的登录流程

玩家登录的时候客户端会首先连接到LoginApp，执行login请求：

void LoginApp::login( const Mercury::Address & source,
		Mercury::UnpackedMessageHeader & header,
		BinaryIStream & data )

这个函数超级长，这里就不贴他的逻辑代码了，大概总结一下这个接口的流程：

1. 判定当前LoginApp是否可对外服务，如果不能对外服务，则拒绝登录
2. 判断当前客户端的ip是否是在黑名单内，如果是则拒绝服务
3. 根据参数里传入的握手协议版本号与服务器的协议版本号进行比较，如果不等则拒绝登录
4. 如果当前客户端已经在登录流程中，则拒绝登录
5. 检查登录限流，这个是通过一个配置项来设定一定时间内能登录的玩家数量，如果当前超过这个数量，则拒绝登录
6. 如果当前LoginApp还没有连接到数据库，则拒绝登录
7. 如果当前LoginApp的负载太高，则拒绝登录
8. 如果当前LoginApp配置了Challenge，则会判断是否需要为这个客户端创建一个Challenge，等待客户端正确的完成这个Challenge之后才能继续后续的流程，这个Challenge一般设计为一种解算时非常消耗CPU但验证时非常容易的问题，典型样例是大质数的分解，服务端随机生成一个256字节的整数，让客户端计算出质因子
9. 开始解析参数里携带的用户名和密码，检查其长度是否过长，如果过长则拒绝登录。这些数据都是使用rsa公钥非对称加密的，只有服务器这里有私钥，所以不用担心泄露
10. 检查客户端是否提供了对称加密的密钥，如果没有提供且当前设置强制要求连接加密，则拒绝登录

当这一切都判断通过之后，会构造一个发送给数据库服务的请求DBAppInterface::logOn，来验证这个登录，验证结果的处理依赖于这里创建的DatabaseReplyHandler：

INFO_MSG( "LoginApp::login: Logging in user '%s' (%s)\n",
	pParams->username().c_str(),
	source.c_str() );

// Remember that this attempt is now in progress and discard further
// attempts from that address for some time after it completes.
ClientLoginRequest & loginRequest = loginRequests_[ source ];
loginRequest.reset();
loginRequest.pChannel( pChannel );
loginRequest.pParams( pParams );

DatabaseReplyHandler * pDBHandler =
	new DatabaseReplyHandler( *this, source, pChannel,
		header.replyID, pParams );

Mercury::Bundle	& dbBundle = this->dbAppAlpha().bundle();
dbBundle.startRequest( DBAppInterface::logOn, pDBHandler );

dbBundle << source << *pParams;

this->dbAppAlpha().send();

当DBApp处理logOn请求时，会判断客户端提供的属性定义系统的摘要是否与当前数据库里的属性定义摘要是否匹配，如果不匹配则代表数据库里的数据客户端是无法正确映射的，因此会拒绝登录：

/**
 *  This method handles a logOn request.
 */
void DBApp::logOn( const Mercury::Address & srcAddr,
        const Mercury::UnpackedMessageHeader & header,
        BinaryIStream & data )
{
    Mercury::Address addrForProxy;
    LogOnParamsPtr pParams = new LogOnParams();

    data >> addrForProxy >> *pParams;

    if (pParams->digest() != this->getEntityDefs().getDigest())
    {
        ERROR_MSG( "DBApp::logOn: Incorrect digest\n" );
        this->sendFailure( header.replyID, srcAddr,
            LogOnStatus::LOGIN_REJECTED_BAD_DIGEST,
            "Defs digest mismatch." );
        return;
    }

    this->logOn( srcAddr, header.replyID, pParams, addrForProxy );
}

接下来的logOn调用代码没有贴出来的价值，大概说一下流程：

1. 如果当前服务器还没有准备好，则拒绝登录
2. 如果当前进程负载比较高，则拒绝登录
3. 如果集群中有任意的CellApp负载比较高，则拒绝登录

全都判断通过之后，创建一个LoginHandler来委托后续的处理：

LoginHandler * pHandler =
	new LoginHandler( pParams, addrForProxy, srcAddr, replyID );

pHandler->login();

这个pHandler->login则由多个异步查询组成，首先是验证用户名和密码是否匹配，这里会去查询账号数据库：

/**
 *  Start the login process
 */
void LoginHandler::login()
{
    DBApp::instance().pBillingSystem()->getEntityKeyForAccount(
        pParams_->username(), pParams_->password(), clientAddr_, *this );

    // When getEntityKeyForAccount() completes, onGetEntityKeyForAccount*()
    // will be called.
}

根据查询结果，会有四个返回分支，这里我们为了简化，忽略掉新建账号和新建角色这两个分支，只考虑下面的两个回调分支：

1. onGetEntityKeyForAccountFailure，代表没有用户或者密码不匹配，直接通知回LoginApp
2. onGetEntityKeyForAccountSuccess，这个代表用户名密码匹配，这里的返回结果里会带上账号对应角色在数据库里的唯一标识符

这里BigWorld的账号与角色数据是一对一映射的，不像我们在mosaic_game中设计的一个账号可以有多个角色数据，所以这里就没有选择角色这一步，直接在查询回来之后以这个EntityKey来查询角色数据库，获取角色的所有属性数据：

/*
 *  IGetEntityKeyForAccountHandler override
 */
void LoginHandler::onGetEntityKeyForAccountSuccess( const EntityKey & ekey,
    const BW::string & dataForClient,
    const BW::string & dataForBaseEntity )
{
    dataForClient_ = dataForClient;
    dataForBaseEntity_ = dataForBaseEntity;
    this->loadEntity( EntityDBKey( ekey ) );
}

/**
 *  This method loads the entity with the given key.
 */
void LoginHandler::loadEntity( const EntityDBKey & ekey )
{
    entityKey_ = ekey;

    // Start "create new base" message even though we're not sure entity
    // exists. This is to take advantage of getEntity() streaming properties
    // into the bundle directly.
    pStrmDbID_ = DBApp::prepareCreateEntityBundle( entityKey_.typeID,
        entityKey_.dbID, clientAddr_, this, bundle_, pParams_,
        &dataForBaseEntity_ );

    // Get entity data
    pBaseRef_ = &baseRef_;

    DBApp::instance().getEntity( ekey, &bundle_, true, *this );
    // When getEntity() completes, onGetEntityCompleted() is called.
}

这里的DBApp::prepareCreateEntityBundle负责开始构造一个往BaseAppMgr投递的CreateEntity的请求数据，但是这个数据还并没有构造完全，因为当前角色的数据还没开始加载。当角色数据加载完成之后，会使用checkOutEntity将玩家数据发送到一个BaseApp上，来创建一个Base对象：

/**
 *  This function checks out the login entity. Must be called after
 *  entity has been successfully retrieved from the database.
 */
void LoginHandler::checkOutEntity()
{
    if ((pBaseRef_ == NULL) &&
        DBApp::instance().onStartEntityCheckout( entityKey_ ))
    {
        // Not checked out and not in the process of being checked out.
        DBApp::setBaseRefToLoggingOn( baseRef_, entityKey_.typeID );
        pBaseRef_ = &baseRef_;

        DBApp::instance().setBaseEntityLocation( entityKey_, baseRef_,
                reserveBaseMailboxHandler_ );
        // When completes, onReservedBaseMailbox() is called.
    }
    else    // Checked out
    {
        DBApp::instance().onLogOnLoggedOnUser( entityKey_.typeID,
            entityKey_.dbID, pParams_, clientAddr_, replyAddr_, replyID_,
            pBaseRef_, dataForClient_, dataForBaseEntity_ );

        delete this;
    }
}

这里checkOutEntity的else分支处理的是顶号的情况，我们目前可以先忽略。DBApp::instance().setBaseEntityLocation负责在数据库中先增加这个角色到所属BaseApp的映射，即使目前的被分配BaseApp是空的。这个操作结束的时候，reserveBaseMailboxHandler_会被调用，通过onReservedBaseMailbox来触发之前填充好的bundle_的发送：

/**
 *  This method is called when the record in bigworldLogOns has been created
 *  or returned.
 */
void LoginHandler::onReservedBaseMailbox( bool isOK, DatabaseID dbID )
{
    if (isOK)
    {
        this->sendCreateEntityMsg();
    }
    else
    {
        DBApp::instance().onCompleteEntityCheckout( entityKey_, NULL );
        // Something horrible like database disconnected or something.
        this->sendFailureReply(
                LogOnStatus::LOGIN_REJECTED_DB_GENERAL_FAILURE,
                "Unexpected database failure." );
    }
}

这里的sendCreateEntityMsg负责最终的createEntity buindle_的send，发送的目标是BaseAppMgr:

/**
 *  This method sends the BaseAppMgrInterface::createEntity message.
 *  Assumes bundle has the right data.
 */
inline void LoginHandler::sendCreateEntityMsg()
{
    INFO_MSG( "DBApp::logOn: %s\n", pParams_->username().c_str() );

    DBApp::instance().baseAppMgr().send( &bundle_ );
}

BaseAppMgr::createEntity里会根据负载均衡来寻找一个负载最低且小于指定阈值的的BaseApp，如果找不到则通知DbApp请求失败，否则调用addEntity接口来转发创建Entity的请求

/**
 *	This method handles the createEntity message. It is called by DBApp when
 *	logging in.
 */
void BaseAppMgr::createEntity( const Mercury::Address & srcAddr,
		const Mercury::UnpackedMessageHeader & header,
		BinaryIStream & data )
{
	Mercury::Address baseAppAddr( 0, 0 );

	BaseApp * pBest = baseApps_.findLeastLoadedApp();

	// 省略一些负载检查的代码

	// Copy the client endpoint address
	baseAppAddr = pBest->externalAddr();

	CreateBaseReplyHandler * pHandler =
		new CreateBaseReplyHandler( srcAddr, header.replyID,
			baseAppAddr );

	// Tell the BaseApp about the client's new proxy
	Mercury::Bundle	& bundle = pBest->bundle();
	bundle.startRequest( BaseAppIntInterface::createBaseWithCellData,
			pHandler );

	bundle.transfer( data, data.remainingLength() );
	pBest->send();

	// Update the load estimate.
	pBest->addEntity();
}

当BaseApp接收到这个createBaseWithCellData的时候，会直接转发到全局的EntityCreator上：

/**
 *	This method creates a base entity on this app. It is used to create a client
 *	proxy or base entities.
 */
void BaseApp::createBaseWithCellData( const Mercury::Address& srcAddr,
		const Mercury::UnpackedMessageHeader& header,
		BinaryIStream & data )
{
	pEntityCreator_->createBaseWithCellData( srcAddr, header, data,
			pLoginHandler_.get() );
}

EntityCreator::createBaseWithCellData这里根据传入的数据来确定要创建的是不带客户端的Base对象还是带客户端的Proxy对象。这里的Proxy是Base的子类，所以createBaseFromStream的返回值是BasePtr：

/**
 *	This method creates a base entity on this app. It is used to create a client
 *	proxy or base entities.
 */
void EntityCreator::createBaseWithCellData( const Mercury::Address& srcAddr,
		const Mercury::UnpackedMessageHeader& header,
		BinaryIStream & data,
		LoginHandler * pLoginHandler )
{
	// TRACE_MSG( "BaseApp::createBaseWithCellData:\n" );

	// The format of the data is as follows:
	// EntityID		id
	// EntityTypeID typeId;
	// DatabaseID	databaseID;
	//
	// For proxy:
	// 	clientAddr
	//#include "address_load_pair.hpp"

	// BASE_DATA
	// CELL_DATA (if needed)
	// Vector3 position (if needed)
	// Vector3 direction (if needed)

	Mercury::Address clientAddr = Mercury::Address::NONE;
	BW::string encryptionKey;

	BasePtr pBase =
		this->createBaseFromStream( data, &clientAddr, &encryptionKey );

	// Replying with an empty response is considered to be a failure report, so
	// any of the early returns from this method will cause this ChannelSender
	// to do just that.
	Mercury::ChannelSender sender( BaseApp::getChannel( srcAddr ) );
	sender.bundle().startReply( header.replyID );

	EntityMailBoxRef ref;
	ref.init();

	if (!pBase)
	{
		sender.bundle() << ref;
		return;
	}

	// Note: If the reply format changes, check that BaseApp::logOnAttempt is
	// okay.
	ref = pBase->baseEntityMailBoxRef();
	sender.bundle() << ref;

	// This is ugly. We should avoid differences in Base and Proxy.
	if (pBase->isProxy() && clientAddr != Mercury::Address::NONE)
	{
		Proxy * pProxy = (Proxy*)pBase.get();
		SessionKey loginKey = pProxy->prepareForLogin( clientAddr );
		sender.bundle() << loginKey;

		if (!encryptionKey.empty())
		{
			pProxy->encryptionKey( encryptionKey );
		}
	}
}

创建完Proxy之后，就开始通知请求的发起方BaseAppMgr当前的CreateEntity操作执行完成，返回的数据里首先填入的就是当前Base对象的通信地址baseEntityMailBoxRef。如果创建的是Proxy且 传入的参数中对应的客户端地址不为空，则调用Proxy::prepareForLogin(clientAddr)来获取一个作为登录SessionId的loginKey，并把该loginKey写入reply bundle中。这里创建loginkey的职责会一路转发，并最终执行到PendingLogins::add：

/**
 *	This method adds a proxy to the set of pending logins. A baseAppLogin
 *	request is now expected from the client to take it out of this set.
 */
SessionKey PendingLogins::add( Proxy * pProxy,
		const Mercury::Address & loginAppAddr )
{
	SessionKey loginKey = pProxy->sessionKey();
	pProxy->regenerateSessionKey();

	// Make sure proxy is only in the pending list once.
	// Note: Brute-force but not too much of an issue here.
	for (iterator iter = container_.begin(); iter != container_.end(); ++iter)
	{
		if (iter->second.pProxy() == pProxy)
		{
			container_.erase( iter );
			break;
		}
	}

	container_.insert( Container::value_type( loginKey,
		PendingLogin( pProxy, loginAppAddr ) ) );

	// Could make this configurable.
	const int PENDING_LOGINS_TIMEOUT = 30; // 30 seconds

	queue_.push_back( QueueElement(
			BaseApp::instance().time() +
				PENDING_LOGINS_TIMEOUT * BaseAppConfig::updateHertz(),
			pProxy->id(), loginKey ) );

	return loginKey;
}

/**
 * When a client logs in, we give a different session key to the login key
 * used to first connect. This one makes a new one, presumably it should only
 * be used soon before the key is sent to the client.
 */
void Proxy::regenerateSessionKey()
{
	do
	{
		// TODO: Not sure why this cannot be 0. If anyone finds out why, write
		// a comment!!
		sessionKey_ = uint32( timestamp() );
	}
	while (sessionKey_ == 0);
}

PendingLogins::add生成loginkey的生成规则其实很简单，就是当前的时间戳。Proxy对象创建的时候就会调用这个regenerateSessionKey来初始化，从而保证不同的客户端利用同一个Proxy来登录的时候其唯一标识符是不一样的。创建完loginkey之后，就会将(loginkey, proxy, loginappaddr)这三元组塞入到等待登录的队列中，同时还会加入到一个默认30s超时的队列来执行自动超时清除操作。

当BaseAppMgr接收到这个CreateEntity的应答返回之后，其对应的CreateBaseReplyHandler负责将这个数据再转发到DbApp上的所属LoginHandler去处理，这里解析出所创建的Proxy的通信地址proxyAddr以及对应的baseRef之后， 会通过DBApp::instance().setBaseEntityLocation将当前entityKey_与baseRef的映射建立出来，这样就记录好了当前账号对应的Proxy对象现在在哪一个BaseApp上：

/*
 *  Mercury::ReplyMessageHandler override.
 */
void LoginHandler::handleMessage( const Mercury::Address & source,
    Mercury::UnpackedMessageHeader & header,
    BinaryIStream & data,
    void * arg )
{
    Mercury::Address proxyAddr;

    data >> proxyAddr;

    if (proxyAddr.ip == 0)
    {
        LogOnStatus::Status status;
        switch (proxyAddr.port)
        {
            case BaseAppMgrInterface::CREATE_ENTITY_ERROR_NO_BASEAPPS:
                status = LogOnStatus::LOGIN_REJECTED_NO_BASEAPPS;
                break;
            case BaseAppMgrInterface::CREATE_ENTITY_ERROR_BASEAPPS_OVERLOADED:
                status = LogOnStatus::LOGIN_REJECTED_BASEAPP_OVERLOAD;
                break;
            default:
                status = LogOnStatus::LOGIN_CUSTOM_DEFINED_ERROR;
                break;
        }

        this->handleFailure( &data, status );
    }
    else
    {
        data >> baseRef_;

        bundle_.clear();
        bundle_.startReply( replyID_ );

        // Assume success.
        bundle_ << (uint8)LogOnStatus::LOGGED_ON;
        bundle_ << proxyAddr;

        // session key
        MF_ASSERT_DEV( data.remainingLength() == sizeof( SessionKey ) );
        bundle_.transfer( data, data.remainingLength() );

        bundle_ << dataForClient_;

        if (entityKey_.dbID != 0)
        {
            pBaseRef_ = &baseRef_;
            DBApp::instance().setBaseEntityLocation( entityKey_,
                    baseRef_, setBaseMailboxHandler_ );
            // When completes, onSetBaseMailbox() is called.
        }
        else
        {
            // Must be "createUnknown", and "rememberUnknown" is false.
            this->sendReply();
        }
    }
}

这里的setBaseEntityLocation作用就是将Proxy对应的MailBox数据记录到数据库里,这里的putExplicitID会被设置为false，同时mailbox的地址也会传递进去:

void setBaseEntityLocation( const EntityKey & entityKey,
		EntityMailBoxRef & mailbox,
		IDatabase::IPutEntityHandler & handler,
		UpdateAutoLoad updateAutoLoad = UPDATE_AUTO_LOAD_RETAIN )
{
	this->putEntity( entityKey, mailbox.id, NULL, &mailbox, 
		false, false,
		updateAutoLoad,
		handler );
}
/**
 *  This method is meant to be called instead of IDatabase::putEntity() so that
 *  we can muck around with stuff before passing it to IDatabase.
 */
void DBApp::putEntity( const EntityKey & entityKey,
        EntityID entityID,
        BinaryIStream * pStream,
        EntityMailBoxRef * pBaseMailbox,
        bool removeBaseMailbox,
        bool putExplicitID,
        UpdateAutoLoad updateAutoLoad,
        IDatabase::IPutEntityHandler& handler )
{
    // Update mailbox for dead BaseApps.
    if (this->hasMailboxRemapping() && pBaseMailbox)
    {
        // Update mailbox for dead BaseApps.
        this->remapMailbox( *pBaseMailbox );
    }

    pDatabase_->putEntity( entityKey, entityID,
            pStream, pBaseMailbox, removeBaseMailbox, 
            putExplicitID, updateAutoLoad, handler );
}

/**
 *	Override from IDatabase
 */
void MySqlDatabase::putEntity( const EntityKey & entityKey,
						EntityID entityID,
						BinaryIStream * pStream,
						const EntityMailBoxRef * pBaseMailbox,
						bool removeBaseMailbox,
						bool putExplicitID,
						UpdateAutoLoad updateAutoLoad,
						IPutEntityHandler & handler )
{
	const EntityTypeMapping * pEntityTypeMapping =
			entityTypeMappings_[ entityKey.typeID ];
	if (pEntityTypeMapping == NULL)
	{
		ERROR_MSG( "MySqlDatabase::putEntity: Entity with id \'%d\' is invalid."
				" Aborting. Please remove from entities.xml or fix def"
				" and script of this entity. ", entityKey.typeID );
		handler.onPutEntityComplete( false, entityKey.dbID );
		return;
	}

	// Note: gameTime is provided to PutEntityTask via the stream
	pBufferedEntityTasks_->addBackgroundTask(
			new PutEntityTask( pEntityTypeMapping,
				entityKey.dbID, entityID,
				pStream, pBaseMailbox, removeBaseMailbox, putExplicitID,
				updateAutoLoad, handler ) );
}

这里会一路传递到底层的MySqlDatabase::putEntity方法，在这个方法里会创建一个PutEntityTask任务，然后将这个任务添加到pBufferedEntityTasks_队列里，等待后续的执行。此时构造函数里会将pBaseMailbox赋值给baseMailbox_，同时writeBaseMailbox_会被设置为true，putExplicitID_会被设置为false，updateAutoLoad_会被设置为UPDATE_AUTO_LOAD_RETAIN, writeEntityData_被设置为false：

/**
 *	Constructor.
 *
 *	Stores all required information so that the task can be executed in a
 *	separate thread.
 */
PutEntityTask::PutEntityTask( const EntityTypeMapping * pEntityTypeMapping,
								DatabaseID databaseID,
								EntityID entityID,
								BinaryIStream * pStream,
								const EntityMailBoxRef * pBaseMailbox,
								bool removeBaseMailbox,
								bool putExplicitID,
								UpdateAutoLoad updateAutoLoad,
								IDatabase::IPutEntityHandler & handler,
								GameTime * pGameTime ) :
	EntityTaskWithID( *pEntityTypeMapping, databaseID, entityID, "PutEntityTask" ),
	writeEntityData_( false ),
	writeBaseMailbox_( false ),
	removeBaseMailbox_( removeBaseMailbox ),
	putExplicitID_( putExplicitID ),
	updateAutoLoad_( updateAutoLoad ),
	handler_( handler ),
	pGameTime_( pGameTime )
{
	if (pStream != NULL)
	{
		stream_.transfer( *pStream, pStream->remainingLength() );
		writeEntityData_ = true;
	}

	if (pBaseMailbox)
	{
		baseMailbox_ = *pBaseMailbox;
		writeBaseMailbox_ = true;
	}
}

当这个PutEntityTask任务被执行的时候，会调用PutEntityTask::performBackgroundTask方法，在这个方法里会执行到writeBaseMailbox_=true的分支，这里会通过entityTypeMapping_.addLogOnRecord方法将Proxy对应的MailBox数据记录到数据库里:

/**
 *	This method writes the entity data into the database.
 */
void PutEntityTask::performBackgroundTask( MySql & conn )
{
	bool definitelyExists = false;

	MF_ASSERT( dbID_ != PENDING_DATABASE_ID );

	if (writeEntityData_)
	{
		// 省略无关代码
	}
	if (writeBaseMailbox_)
	{
		// Check for existence to prevent adding invalid LogOn records
		if (definitelyExists ||
			entityTypeMapping_.checkExists( conn, dbID_ ))
		{
			// Add or update the log on record.
			entityTypeMapping_.addLogOnRecord( conn,
					dbID_, baseMailbox_ );
		}
	}
	// 省略一些代码
}

这个EntityTypeMapping::addLogOnRecord其实就是将当前Proxy的MailBox数据记录到数据库里, 表为bigworldLogOns:

namespace
{
const Query addLogOnQuery(
	"INSERT INTO bigworldLogOns "
			"(databaseID, typeID, objectID, ip, port, salt) "
		"VALUES (?,?,?,?,?,?) "
		"ON DUPLICATE KEY "
		"UPDATE "
			"objectID = VALUES(objectID), "
			"ip = VALUES(ip), "
			"port = VALUES(port), "
			"salt = VALUES(salt)" );
}
/**
 *
 */
void EntityTypeMapping::addLogOnRecord( MySql & connection,
			DatabaseID dbID, const EntityMailBoxRef & mailbox ) const
{
	addLogOnQuery.execute( connection,
			dbID, this->getDatabaseTypeID(),
			mailbox.id,
			htonl( mailbox.addr.ip ),
			htons( mailbox.addr.port ),
			mailbox.addr.salt,
			NULL );
}

这个数据库任务执行完成之后，会一路回调，调用到setBaseEntityLocation时设置好的setBaseMailboxHandler_，这里会调用到LoginHandler::onSetBaseMailbox方法，内部将之前构造好的reply发送回LoginApp:

/**
 *  This method is called when the record in bigworldLogOns has been set.
 */
void LoginHandler::onSetBaseMailbox( bool isOK, DatabaseID dbID )
{
    DBApp::instance().onCompleteEntityCheckout( entityKey_,
            isOK ? &baseRef_ : NULL );

    if (isOK)
    {
        this->sendReply();
    }
    else
    {
        // Something horrible like database disconnected or something.
        this->sendFailureReply(
                LogOnStatus::LOGIN_REJECTED_DB_GENERAL_FAILURE,
                "Unexpected database failure." );
    }
}

这个Reply的内容就是Proxy的通信地址以及对应的LoginKey。不过当LoginApp接收到这个返回消息的时候，并不是分开解析这两个字段，而是使用一个LoginReplyRecord来封装一下：

/**
 * 	This structure contains the reply from a successful login.
 */
struct LoginReplyRecord
{
	Mercury::Address	serverAddr;			// send to here
	uint32				sessionKey;			// use this session key
};


/**
 *  This method is called when a message comes back from the system.
 *  It deletes itself at the end.
 */
void DatabaseReplyHandler::handleMessage(
    const Mercury::Address & /*source*/,
    Mercury::UnpackedMessageHeader & header,
    BinaryIStream & data,
    void * /*arg*/ )
{
    uint8 status;
    data >> status;

    if (status != LogOnStatus::LOGGED_ON)
    {
        // 忽略许多错误处理
        delete this;
        return;
    }

    if (data.remainingLength() < int(sizeof( LoginReplyRecord )))
    {
        // 忽略一些错误处理

        delete this;
        return;
    }

    LoginReplyRecord lrr;
    data >> lrr;

    BW::string serverMsg;

    if (data.remainingLength() > 0)
    {
        data >> serverMsg;
    }

	// 省略网络NAT映射的代码

    loginApp_.sendAndCacheSuccess( clientAddr_, pChannel_.get(),
            replyID_, lrr, serverMsg, pParams_ );

    delete this;
}

这里的sendAndCacheSuccess会将当前的登录信息存储在loginRequests_这个map上，key为客户端地址，value为一个ClientLoginRequest。填充好这个ClientLoginRequest之后，调用sendSuccess将登录成功的数据下发到客户端：

/**
 *	This method sends a reply to a client indicating that logging in has been
 *	successful. It also caches this information so that it can be resent if
 *	necessary.
 */
void LoginApp::sendAndCacheSuccess( const Mercury::Address & addr,
		Mercury::Channel * pChannel, Mercury::ReplyID replyID,
		const LoginReplyRecord & replyRecord,
		const BW::string & serverMsg, LogOnParamsPtr pParams )
{
	ClientLoginRequest & request = loginRequests_[ addr ];
	request.setData( replyRecord, serverMsg );

	MF_ASSERT_DEV( *pParams == *request.pParams() );

	this->sendSuccess( addr, pChannel, replyID, request );

	// Do not let the map get too big. Just check every so often to get rid of
	// old caches.

	// 省略一些清除过期的loginrequests的代码
}

sendSuccess这里往下发送成功的数据的时候，会使用客户端传递过来的对称加密密钥encryptionKey来进行数据加密，加密之后再调用sendRawReply发送到客户端：

/**
 *	This method sends a reply to a client indicating that logging in has been
 *	successful.
 */
void LoginApp::sendSuccess( const Mercury::Address & addr,
		Mercury::Channel * pChannel, Mercury::ReplyID replyID,
		const ClientLoginRequest & request )
{
	MemoryOStream data;

	data << (int8)LogOnStatus::LOGGED_ON;
	
	const BW::string & encryptionKey = request.pParams()->encryptionKey();

	if (!encryptionKey.empty())
	{
		// We have to encrypt the reply record because it contains the session
		// key
		Mercury::EncryptionFilterPtr pFilter =
			Mercury::EncryptionFilter::create( 
				Mercury::SymmetricBlockCipher::create( encryptionKey ) );
		MemoryOStream clearText;
		request.writeSuccessResultToStream( clearText );
		pFilter->encryptStream( clearText, data );
	}
	else
	{
		request.writeSuccessResultToStream( data );
	}

	loginStats_.incSuccesses();
	++gNumLogins;

	this->sendRawReply( addr, pChannel, replyID, data );
}

当loginApp_.sendAndCacheSuccess执行完之后，LoginApp就会删除当前的DatabaseReplyHandler，至此LoginApp的登录流程彻底结束。

当客户端收到这个登录成功的消息之后，客户端就拿到了对应的Proxy的通信地址以及登录的会话标识符LoginKey，这样客户端就会发送一个baseAppLogin登录请求到这个Proxy:

/**
 *	This method handles a message from the client. Should be the first message
 *	received from the client.
 */
void BaseApp::baseAppLogin( const Mercury::Address& srcAddr,
			const Mercury::UnpackedMessageHeader& header,
			const BaseAppExtInterface::baseAppLoginArgs & args )
{
	pLoginHandler_->login( extInterface_, srcAddr, header, args );
}

/**
 *	This method is called by a client to make initial contact with the BaseApp.
 *	It should be called after the client has logged in via the LoginApp.
 */
void LoginHandler::login( Mercury::NetworkInterface & networkInterface,
			const Mercury::Address& srcAddr,
			const Mercury::UnpackedMessageHeader& header,
			const BaseAppExtInterface::baseAppLoginArgs & args )
{
	PendingLogins::iterator pendingIter = pPendingLogins_->find( args.key );

	if (pendingIter == pPendingLogins_->end())
	{
		INFO_MSG( "LoginHandler::login(%s): "
				"No pending login for loginKey %u. Attempt = %u\n",
			srcAddr.c_str(), args.key, args.numAttempts );
		// Bad bundle so break out of dispatching the rest.
		header.breakBundleLoop();

		return;
	}

	const PendingLogin & pending = pendingIter->second;

	SmartPointer<Proxy> pProxy = pending.pProxy();

	if (pProxy->isDestroyed())
	{
		return;
	}

	if (networkInterface.findChannel( srcAddr ) != NULL)
	{
		++numLoginCollisions_;

		INFO_MSG( "LoginHandler::login(%s): "
				"%u collided with an existing channel. Attempt = %u\n",
			srcAddr.c_str(), pProxy->id(), args.numAttempts );

		return;
	}

	this->updateStatistics( srcAddr, pending.addrFromLoginApp(), args.numAttempts );

	pPendingLogins_->erase( pendingIter );

	if (pProxy->attachToClient( srcAddr, header.replyID,
			header.pChannel.get() ))
	{
		INFO_MSG( "LoginHandler::login: "
			"%u attached from %s. Attempt %u\n",
			pProxy->id(), srcAddr.c_str(), args.numAttempts );
	}
}

这里会解析出args.key，也就是我们之前在proxy创建成功之后记录的LoginKey, 通过这个LoginKey来查询所关联的Proxy对象，最终使用attachToClient来绑定客户端的地址到当前Proxy的客户端信道pChannel上：

/**
 *	This method attaches this proxy to the client at the input address.
 *	It is only used the first time this client is attached to a proxy on this
 *	BaseApp, local handoffs are entirely handled by @see giveClientTo.
 *
 *	@param clientAddr 		The client address.
 *	@param loginReplyID		The reply ID of the login request message.
 *	@param pChannel			If this is a Mercury/TCP request, the associated
 *							TCP channel, otherwise NULL.
 */
bool Proxy::attachToClient( const Mercury::Address & clientAddr,
		Mercury::ReplyID loginReplyID, 
		Mercury::Channel * pChannel )
{
	AUTO_SCOPED_THIS_ENTITY_PROFILE;

	IF_NOT_MF_ASSERT_DEV( pClientChannel_ == NULL )
	{
		return false;
	}

	BaseApp & baseApp = BaseApp::instance();

	bool hasClient = (clientAddr != Mercury::Address::NONE);

	if (hasClient)
	{
		// Create the BlockCipher to encrypt the channel with.
		Mercury::BlockCipherPtr pBlockCipher = NULL;

		if (encryptionKey_.empty())
		{
			ERROR_MSG( "Proxy::attachToClient( %s ): "
					"No session encryption key, falling back to unencrypted "
					"connection\n",
				clientAddr.c_str() );
		}
		else
		{
			pBlockCipher = 
				Mercury::SymmetricBlockCipher::create( encryptionKey_ );

			if (!pBlockCipher)
			{
				ERROR_MSG( "Proxy::attachToClient( %s ): "
						"Invalid encryption key, falling back to unencrypted "
						"connection\n",
					clientAddr.c_str() );
			}
		}

		if (pChannel == NULL)
		{
			// TCP Channels have already been created before we process
			// bundles, UDP channels need to be created explicitly.

			pChannel = new Mercury::UDPChannel(
				baseApp.extInterface(),
				clientAddr,
				Mercury::UDPChannel::EXTERNAL,
				MIN_CLIENT_INACTIVITY_RESEND_DELAY );
		}

		if (pBlockCipher)
		{
			pChannel->setEncryption( pBlockCipher );
		}

		this->setClientChannel( pChannel );

		// 省略一些代码

		// now we are ready for the world to know about us.
		baseApp.addProxy( this );

		// create an object to push ourselves internally
		MF_ASSERT( pProxyPusher_ == NULL );
		if (!this->hasCellEntity())
		{
			pProxyPusher_ = new ProxyPusher( this );
		}

		// Send the login reply before anything else (if required).  This must
		// be the first message on this channel and must be on a bundle by
		// itself.  Even though the login message is off-channel (because it has
		// to be - we don't know the client's address until we get it), we want
		// to send back the reply on the channel because the client has a
		// channel for us now and if we send this off-channel the PacketFilters
		// won't work.  Also - this means that all downstream traffic to the
		// client is filtered (i.e. encrypted).
		if (loginReplyID != Mercury::REPLY_ID_NONE)
		{
			// Make a new session key to send with the reply.
			// Don't buffer this messages behind createBasePlayer
			Mercury::Bundle & bundle = pClientChannel_->bundle();
			bundle.startReply( loginReplyID );
			bundle << sessionKey_;
			this->sendBundleToClient();
		}

		// Now that the external interface will have the ClientInterface
		// registered, we can prime bundles with those messages.
		pClientChannel_->bundlePrimer( &clientBundlePrimer_ );

		// Don't buffer these messages behind createBasePlayer
		Mercury::Bundle & b = pClientChannel_->bundle();

		ClientInterface::updateFrequencyNotificationArgs & frequencyArgs =
			ClientInterface::updateFrequencyNotificationArgs::start( b );

		frequencyArgs.hertz = uint8(BaseAppConfig::updateHertz());

		ClientInterface::setGameTimeArgs::start( b ).gameTime = baseApp.time();

		// 省略一些代码
	}
	else
	{
		INFO_MSG( "Proxy::attachToClient: "
			"Channel not created for %u. No client yet.\n", id_ );
	}

	return true;
}

至此，客户端的登录流程彻底结束，整体流程图参见下图：

客户端现在知道了对应Proxy的通信地址，同时Proxy也根据clientAddr创建了一个UDPChannel并设置到了pChannel_上，两者之间的通信都使用了一个对称加密密钥encryptionKey_来加密。

客户端连接确立之后，整个信道就建立了，消息可以这个信道中按序收发，直到异常状况发生。这些异常情况主要有三类:主动下线、超时掉线和顶号登录，接下来将逐个介绍这三种情况的处理流程。

客户端的下线流程

如果客户端想要通知服务器当前当前角色需要主动下线，那么客户端会发送一个disconnectClient消息到服务器，服务器收到这个消息之后会主动调用Proxy::disconnectClient函数来处理这个请求,此时第一个参数reason被填充为CLIENT_DISCONNECT_CLIENT_REQUESTED:

/**
 *	This method handles a message from the client telling us that we should
 *	disconnect it.
 */
void Proxy::disconnectClient(
		const BaseAppExtInterface::disconnectClientArgs & args )
{
	this->onClientDeath( CLIENT_DISCONNECT_CLIENT_REQUESTED );
}

上面的函数onClientDeath会根据channel查找绑定的Proxy，通知其客户端断线，这里会把逻辑处理委托到logOffClient函数上:

/**
 *	This gets called when a client dies. Currently, we use it to not send them
 *	any more packets. It also informs the client's relatives (the Cell), which
 *	grieves for a few microseconds then removes it.
 */
void Proxy::onClientDeath( ClientDisconnectReason reason,
		bool shouldExpectClient /* = true */ )
{
	AUTO_SCOPED_THIS_ENTITY_PROFILE;

	TRACE_MSG( "Proxy::onClientDeath: Client id %u has disconnected (%s)\n", 
		this->id(), clientDisconnectReasonToString( reason ) );

	// we can be called again if we are already dead if we try to send more
	// stuff to the client ... for now we just ignore it but could do no more...
	if (!this->hasClient() && shouldExpectClient)
	{
		return;	// already dead and told the cell about it
	}

	if (shouldExpectClient || this->hasClient())
	{
		// If we don't expect a client, we don't care about finalising
		// acks from the channel
		bool shouldCondemnClient = shouldExpectClient;

		if (reason == CLIENT_DISCONNECT_TIMEOUT ||
			reason == CLIENT_DISCONNECT_RATE_LIMITS_EXCEEDED )
		{
			// We don't care about finalising acks from a timed-out channel
			shouldCondemnClient = false;
		}
		this->logOffClient( shouldCondemnClient );
	}

	// OK, we know the cell and we haven't told it about it yet. Call away
	PyObject * pFunc = PyObject_GetAttrString( this, "onClientDeath" );

	// 这里会通知脚本对象执行onClientDeath回调
}

logOffClient需要一个参数shouldCondemnClient，表示在断开客户端连接时是否应该保留该底层通道，而不是立即销毁它,在主动下线时这个参数为true：

/**
 *  Send the client a disconnect message, then disconnect.
 *
 *	@param shouldCondemnChannel	True if any client channel should be condemned,
 *								false if it should be immediately destroyed.
 *								Generally true unless it was a client time-out
 *								or other situation where the client channel is
 *								probably no longer in a good state.
 */
void Proxy::logOffClient( bool shouldCondemnChannel )
{
	AUTO_SCOPED_THIS_ENTITY_PROFILE;

	// Abort any pending downloads, and then tell the script about it after
	// the client has gone.
	DownloadCallbacks callbacks;
	dataDownloads_.abortDownloads( callbacks );

	if (this->isClientConnected())
	{
		// Send a message to the client telling it that it has been logged off.
		// The reason parameter is not yet used.
		Mercury::Bundle & bundle = pClientChannel_->bundle();
		ClientInterface::loggedOffArgs::start( bundle ).reason = 0;
		this->sendBundleToClient();
	}

	this->detachFromClient( shouldCondemnChannel );

	callbacks.triggerCallbacks( this );
}

这个shouldCondemnChannel参数会顺带的传递到detachFromClient接口中,来做真正的断线处理:

1. 如果为true：会调用channel->shutDown(),这里会保留通道让它继续发送/接收剩余的确认（ACKs）、完成待发的可靠数据，然后再关闭（优雅断开，允许尾包/重传完成）。
2. 如果为false：会调用channel->destroy() ,立即销毁通道，不再等 ACK和重传（适用于超时、被速率限制或通道状态不可用的情况）。

/**
 *  Finalise a session with the currently attached client.
 *
 *	@param shouldCondemn 	If true, the client channel is condemned (and so
 *				will continue to send acknowledgements for some
 *				time), otherwise, the channel is reset.
 */
void Proxy::detachFromClient( bool shouldCondemn )
{
	isGivingClientAway_ = false;

	if (this->hasClient())
	{
		BaseApp::instance().removeProxy( this );
	}

	if (pClientChannel_ != NULL)
	{
		// Put aside the pointer which setClientChannel
		// uses and then sets to NULL, so we can clean it up.
		pClientChannel_->pChannelListener( NULL );

		Mercury::ChannelPtr pSavedClientChannel = pClientChannel_;
		this->setClientChannel( NULL );

		if (pSavedClientChannel->isConnected())
		{
			if (shouldCondemn)
			{
				pSavedClientChannel->shutDown();
			}
			else
			{
				pSavedClientChannel->destroy();	
			}
		}
	}

	// Don't try to disable the witness if we've already sent the
	// destroyCell message.
	if (cellHasWitness_ && this->shouldSendToCell())
	{
		this->sendEnableDisableWitness( /*enable:*/false );
	}
	cellHasWitness_ = false;

	pBufferedClientBundle_.reset();
	// 省略很多代码
}

原来的pClientChannel_会被设置为nullptr，同时如果在CellApp上有了对应的RealEntity，则通过sendEnableDisableWitness(false)来通知这个RealEntity停止向客户端发送任何消息。

在Proxy::onClientDeath调用完logOffClient后，Proxy的Python脚本那边会调用onClientDeath回调，来通知脚本对象客户端断开连接,此时脚本层可以调用self.destroy()来调用Base::py_destroy来销毁Proxy对象:

PY_KEYWORD_METHOD_DECLARE( py_destroy )

PY_METHOD( destroy )

/**
 *	This method destroys this object when the script says so.
 */
PyObject * Base::py_destroy( PyObject * args, PyObject * kwargs )
{
	if (PyTuple_Size( args ) != 0)
	{
		PyErr_SetString( PyExc_TypeError, "Only expecting keyword arguments" );
		return NULL;
	}

	if (this->hasCellEntity() || this->isGetCellPending())
	{
		PyErr_SetString( PyExc_ValueError,
				"Still has cell entity. Use Base.destroyCellEntity" );

		return NULL;
	}

	if (isDestroyed_)
	{
		PyErr_SetString( PyExc_ValueError, "Base entity already destroyed" );
		return NULL;
	}

	static char * keywords[] = 
	{
		const_cast< char *> ( "deleteFromDB" ),
		const_cast< char *> ( "writeToDB" ),
		NULL
	};

	PyObject * pDeleteFromDB = NULL;
	PyObject * pWriteToDB = NULL;

	if (!PyArg_ParseTupleAndKeywords( args, kwargs, "|OO", keywords,
			&pDeleteFromDB, &pWriteToDB ))
	{
		return NULL;
	}

	bool deleteFromDB = (pDeleteFromDB != NULL) ? 
		PyObject_IsTrue( pDeleteFromDB ) : false;

	bool writeToDB = (pWriteToDB != NULL) ? 
		PyObject_IsTrue( pWriteToDB ) : this->hasWrittenToDB();

	if (pWriteToDB && !writeToDB && BaseApp::instance().pSqliteDB())
	{
		// Writes lost due to flip-floping
		SECONDARYDB_WARNING_MSG( "Base::py_destroy: %s %d destroyed with "
			"writeToDB=False. All writes to the secondary "
			"database will be lost.\n",	pType_->name(), id_ );
	}

	this->destroy( deleteFromDB, writeToDB );

	Py_RETURN_NONE;
}

当Proxy对象上的destory接口被调用时，参数logOffFromDB参数默认为true,此时会设置WriteDBFlags中的WRITE_LOG_OFF标志。 接下来调用Base::writeToDB(flags)，其中flags包含WRITE_LOG_OFF标志:

void destroy( bool deleteFromDB, bool writeToDB, bool logOffFromDB = true );
/**
 *	This method destroys this base.
 */
void Base::destroy( bool deleteFromDB, bool writeToDB, bool logOffFromDB )
{
	IF_NOT_MF_ASSERT_DEV( !isDestroyed_ )
	{
		return;
	}

	if (inDestroy_)
	{
		return;
	}

	inDestroy_ = true;

	Script::call( PyObject_GetAttrString( this, "onDestroy" ),
					PyTuple_New( 0 ), "onDestroy", true );

	// TRACE_MSG( "Base(%d)::destroy: deleteFromDB=%d, writeToDB=%d\n",
	//	id_, deleteFromDB, writeToDB );

	keepAliveTimerHandle_.cancel();

	// Inform our backup that we've been destroyed.
	const Mercury::Address backupAddr =
		BaseApp::instance().backupAddrFor( this->id() );

	if (backupAddr != Mercury::Address::NONE)
	{
		Mercury::ChannelSender sender( BaseApp::getChannel( backupAddr ) );
		BaseAppIntInterface::stopBaseEntityBackupArgs::start(
			sender.bundle() ).entityID = this->id();
	}

	if (this->hasWrittenToDB())
	{
		WriteDBFlags flags = 0;

		if (logOffFromDB)
		{
			flags |= WRITE_LOG_OFF;
		}
		if (deleteFromDB)
		{
			flags |= WRITE_DELETE_FROM_DB;
		}
		else if (writeToDB)
		{
			flags |= WRITE_BASE_CELL_DATA;
		}

		this->writeToDB( flags );
	}

	BaseApp::instance().pGlobalBases()->onBaseDestroyed( this );

	this->discard();

	inDestroy_ = false;
}

这个writeToDB接口会根据flags来判断是否需要写入数据库。如果flags中包含WRITE_LOG_OFF标志，则此时存库消息的回调会被设置为LogOffReplyHandler：

/**
 *	This method writes the entity to the database.
 */
bool Base::writeToDB( WriteDBFlags flags, WriteToDBReplyStructPtr pReplyStruct,
		PyObjectPtr pCellData, DatabaseID explicitDatabaseID )
{
	// 省略很多代码
		bool isLogOff =(flags & WRITE_LOG_OFF);

	if (!shouldWriteToSecondary || isLogOff)
	{
		BaseApp & baseApp = BaseApp::instance();

		DatabaseID dbID = (flags & WRITE_EXPLICIT_DBID) ? 
				explicitDatabaseID : databaseID_;

		if (dbID == PENDING_DATABASE_ID)
		{
			DEBUG_MSG( "Base::writeToDB: %s %d is still pending initial "
					"database write.\n",
				this->pType()->name(), id_ );
			// Set the dbID to 0, which will distribute this operation to
			// the Alpha DBApp.
			dbID = 0;
		}

		const DBAppGateway & dbApp = baseApp.dbAppGatewayFor( dbID );
		
		if (dbApp.address().isNone())
		{
			ERROR_MSG( "Base::writeToDB: No DBApp is available, "
				"data for %" PRI64 " has been lost\n", dbID );
			return false;
		}

		Mercury::Channel & channel = baseApp.getChannel( dbApp.address() );
		std::auto_ptr< Mercury::Bundle > pBundle( channel.newBundle() );

		BinaryOStream * pStream = pBundle.get();

		bool shouldSetToPending = false;

		// We expect a reply if we are getting a database id. That is, this is
		// the first time that we are being written to the database.
		if (pReplyStruct->expectsReply() ||
			(!isLogOff && !this->hasWrittenToDB()))
		{
			shouldSetToPending = (databaseID_ == 0);
			pBundle->startRequest( DBAppInterface::writeEntity,
				new WriteEntityReplyHandler( this, pReplyStruct ) );
		}
		else if (isLogOff && this->hasWrittenToDB())
		{
			// Owned by LogOffReplyHandler
			MemoryOStream * pMemoryStream = new MemoryOStream;
			pStream = pMemoryStream;
			pBundle->startRequest( DBAppInterface::writeEntity,
				new LogOffReplyHandler( this->id(), pMemoryStream ) );
		}
		else
		{
			pBundle->startMessage( DBAppInterface::writeEntity );
		}

		if (flags & WRITE_EXPLICIT_DBID)
		{
			MF_ASSERT_DEV( (flags & WRITE_EXPLICIT_DBID) && (databaseID_ == 0) );
			MF_ASSERT_DEV( (flags & WRITE_EXPLICIT_DBID) && (explicitDatabaseID != 0) );
		}

		*pStream << flags << this->pType()->id() << dbID << this->id();

		if (shouldSetToPending)
		{
			// This needs to be done after databaseID_ is streamed on.
			databaseID_ = PENDING_DATABASE_ID;
		}

		if (!this->addToStream( flags, *pStream, pCellData ))
		{
			return false;
		}

		if (flags & WRITE_BASE_CELL_DATA)
		{
			*pStream << BaseApp::instance().time();
		}

		// If the stream is not already on the bundle, place it on now.
		if (pStream != pBundle.get())
		{
			pBundle->addBlob(
				static_cast< MemoryOStream * >( pStream )->data(),
				static_cast< MemoryOStream * >( pStream )->size() );
		}

		persistentSize = pBundle->size();

		channel.send( pBundle.get() );
	}
}

DBApp::writeEntity处理这个writeEntity请求时,会继续将这个flags传递到WriteEntityHandler里，这个Handler负责处理当前writeEntity请求的具体执行：

/**
 *  This method handles the writeEntity mercury message.
 */
void DBApp::writeEntity( const Mercury::Address & srcAddr,
        const Mercury::UnpackedMessageHeader & header,
        BinaryIStream & data )
{
    AUTO_SCOPED_PROFILE( "writeEntity" );

    WriteDBFlags flags;
    data >> flags;
    // if this fails then the calling component had no need to call us
    MF_ASSERT( flags &
            (WRITE_BASE_CELL_DATA | WRITE_LOG_OFF | WRITE_AUTO_LOAD_MASK) );

    EntityDBKey ekey( 0, 0 );
    data >> ekey.typeID >> ekey.dbID;

    // TRACE_MSG( "DBApp::writeEntity: %lld flags=%i\n",
    //         ekey.dbID, flags );

    bool isOkay = this->getEntityDefs().isValidEntityType( ekey.typeID );

    if (!isOkay)
    {
        ERROR_MSG( "DBApp::writeEntity: Invalid entity type %d\n",
            ekey.typeID );

        if (header.replyID != Mercury::REPLY_ID_NONE)
        {
            Mercury::ChannelSender sender( DBApp::getChannel( srcAddr ) );
            sender.bundle().startReply( header.replyID );
            sender.bundle() << isOkay << ekey.dbID;
        }
    }
    else
    {
        EntityID entityID;
        data >> entityID;

        WriteEntityHandler* pHandler =
            new WriteEntityHandler( ekey, entityID, flags,
                header.replyID, srcAddr );

        if (flags & WRITE_DELETE_FROM_DB)
        {
            pHandler->deleteEntity();
        }
        else
        {
            pHandler->writeEntity( data, entityID );
        }
    }
}

WriteEntityHandler::writeEntity执行存库操作的时候，如果发现flags中包含WRITE_LOG_OFF标志，则调用putEntity方法时的第四个参数removeBaseMailbox为true，表示需要删除该实体的BaseMailbox对象:

/**
 *	This method writes the entity data into the database.
 *
 *	@param	data	Stream should be currently at the start of the entity's
 *	data.
 *	@param	entityID	The entity's base mailbox object ID.
 */
void WriteEntityHandler::writeEntity( BinaryIStream & data, EntityID entityID )
{
	BinaryIStream * pStream = NULL;

	UpdateAutoLoad updateAutoLoad = 
		(flags_ & WRITE_AUTO_LOAD_YES) 	? 	UPDATE_AUTO_LOAD_TRUE :
		(flags_ & WRITE_AUTO_LOAD_NO) 	? 	UPDATE_AUTO_LOAD_FALSE:
											UPDATE_AUTO_LOAD_RETAIN;

	if (flags_ & WRITE_BASE_CELL_DATA)
	{
		pStream = &data;
	}

	if (flags_ & WRITE_LOG_OFF)
	{
		this->putEntity( pStream, updateAutoLoad,
			/* pBaseMailbox: */ NULL,
			/* removeBaseMailbox: */ true );
	}
	else if (ekey_.dbID == 0 ||(flags_ & WRITE_EXPLICIT_DBID))
	{
		// New entity is checked out straight away
		baseRef_.init( entityID, srcAddr_, EntityMailBoxRef::BASE,
			ekey_.typeID );
		this->putEntity( pStream, updateAutoLoad, &baseRef_ );
	}
	else
	{
		this->putEntity( pStream, updateAutoLoad );
	}
	// When putEntity() completes onPutEntityComplete() is called.
}

/**
 *	This method is invoked by WriteEntityHandler::writeEntity to pass through
 *	a putEntity request to the database implementation.
 */
void WriteEntityHandler::putEntity( BinaryIStream * pStream,
			UpdateAutoLoad updateAutoLoad,
			EntityMailBoxRef * pBaseMailbox,
			bool removeBaseMailbox )
{
	DBApp::instance().putEntity( ekey_, entityID_,
			pStream, pBaseMailbox, removeBaseMailbox, 
			flags_ & WRITE_EXPLICIT_DBID, 
			updateAutoLoad, *this );
}

/**
 *  This method is meant to be called instead of IDatabase::putEntity() so that
 *  we can muck around with stuff before passing it to IDatabase.
 */
void DBApp::putEntity( const EntityKey & entityKey,
        EntityID entityID,
        BinaryIStream * pStream,
        EntityMailBoxRef * pBaseMailbox,
        bool removeBaseMailbox,
        bool putExplicitID,
        UpdateAutoLoad updateAutoLoad,
        IDatabase::IPutEntityHandler& handler )
{
    // Update mailbox for dead BaseApps.
    if (this->hasMailboxRemapping() && pBaseMailbox)
    {
        // Update mailbox for dead BaseApps.
        this->remapMailbox( *pBaseMailbox );
    }

    pDatabase_->putEntity( entityKey, entityID,
            pStream, pBaseMailbox, removeBaseMailbox, 
            putExplicitID, updateAutoLoad, handler );
}


/**
 *	Override from IDatabase
 */
void MySqlDatabase::putEntity( const EntityKey & entityKey,
						EntityID entityID,
						BinaryIStream * pStream,
						const EntityMailBoxRef * pBaseMailbox,
						bool removeBaseMailbox,
						bool putExplicitID,
						UpdateAutoLoad updateAutoLoad,
						IPutEntityHandler & handler )
{
	const EntityTypeMapping * pEntityTypeMapping =
			entityTypeMappings_[ entityKey.typeID ];
	if (pEntityTypeMapping == NULL)
	{
		ERROR_MSG( "MySqlDatabase::putEntity: Entity with id \'%d\' is invalid."
				" Aborting. Please remove from entities.xml or fix def"
				" and script of this entity. ", entityKey.typeID );
		handler.onPutEntityComplete( false, entityKey.dbID );
		return;
	}

	// Note: gameTime is provided to PutEntityTask via the stream
	pBufferedEntityTasks_->addBackgroundTask(
			new PutEntityTask( pEntityTypeMapping,
				entityKey.dbID, entityID,
				pStream, pBaseMailbox, removeBaseMailbox, putExplicitID,
				updateAutoLoad, handler ) );
}

这个PutEntityTask在执行任务的时候，如果发现removeBaseMailbox为true，则会执行entityTypeMapping_.removeLogOnRecord方法来从bigworldLogOns表里删除该实体的BaseMailbox记录，刚好对应了登录时的setBaseEntityLocation所添加的记录：

else if (removeBaseMailbox_)
{
	entityTypeMapping_.removeLogOnRecord( conn, dbID_ );
}
namespace
{
const Query removeLogOnQuery(
			"DELETE FROM bigworldLogOns WHERE databaseID = ? AND typeID = ?" );
}

void EntityTypeMapping::removeLogOnRecord( MySql & conn, DatabaseID id ) const
{
	removeLogOnQuery.execute( conn, id, this->getDatabaseTypeID(), NULL );
}

客户端的掉线流程

BigWorld中客户端连接使用的是无连接的UDP而不是有连接的TCP，所以并没有断线这一说。但是可靠UDP在逻辑层模拟了一个可靠连接，当发出的数据长时间没有收到ACK回复的时候，这个通道就会因为超时而被设置为不可用，执行销毁操作：

/**
 *	This method "destroys" this channel. It should be considered similar to
 *	delete pChannel except that there may be other references remaining.
 */
void Channel::destroy()
{
	IF_NOT_MF_ASSERT_DEV( !isDestroyed_ )
	{
		return;
	}

	inactivityTimerHandle_.cancel();
	this->doDestroy();

	isDestroyed_ = true;

	pNetworkInterface_->onChannelGone( this );

	if (this->pChannelListener())
	{
		this->pChannelListener()->onChannelGone( *this );
	}

	this->decRef();
}

销毁时会通知绑定在其身上的channelListener来通知onChannelGone。而在Proxy绑定到一个客户端的Channel的时候， 会执行BaseApp:addProxy，这里会将当前的BaseApp注册为这个通道的ChannelListener：

/**
 *	This method adds a proxy from this manager.
 */
void BaseApp::addProxy( Proxy * pNewProxy )
{
	Mercury::ChannelPtr pChannel = pNewProxy->pClientChannel();
	Mercury::Address address = pChannel->addr();
	address.salt = (pChannel->isTCP() ? 1 : 0);

	TRACE_MSG( "BaseApp: Adding proxy %u at %s\n",
		pNewProxy->id(), pChannel->c_str() );
	// set ourselves in the map from ip address to proxy

	Proxy *& rpProxy = proxies_[address];
	// 省略一些代码

	pChannel->pChannelListener( this );
}

所以当这个客户端通道状态变成超时断线时，BaseApp::onChannelGone就会被执行：

/*
 *	Override from Mercury::ChannelListener.
 */
void BaseApp::onChannelGone( Mercury::Channel & channel )
{
	TRACE_MSG( "BaseApp::onChannelGone: %s\n", channel.c_str() );

	Proxies::iterator iProxy = proxies_.find( channel.addr() );

	if (iProxy != proxies_.end())
	{
		ProxyPtr pProxy = iProxy->second;
		pProxy->onClientDeath( CLIENT_DISCONNECT_TIMEOUT );
	}
}

上面的函数会根据channel查找绑定的Proxy，使用onClientDeath通知其客户端断线。这个onClientDeath已经在前面的小节里介绍过了，与之前的主动下线的差别是这里的shouldCondemnClient参数被设置为false，所以这里会直接销毁Channel，而不是等待ACK：

if (shouldExpectClient || this->hasClient())
{
	// If we don't expect a client, we don't care about finalising
	// acks from the channel
	bool shouldCondemnClient = shouldExpectClient;

	if (reason == CLIENT_DISCONNECT_TIMEOUT ||
		reason == CLIENT_DISCONNECT_RATE_LIMITS_EXCEEDED )
	{
		// We don't care about finalising acks from a timed-out channel
		shouldCondemnClient = false;
	}
	this->logOffClient( shouldCondemnClient );
}

客户端的顶号流程

从上述流程代码可以看出，一旦客户端连接超时，则Proxy会通过detachFromClient将此Channel直接销毁，抛弃当前未ACK以及后续的所有发往客户端的消息。所以这里并没有mosaic_game中的断线重连机制，只有断线。那客户端怎么重新连接到服务器呢，答案是走重新登录顶号机制。

重新登录时，客户端发送的数据与常规登录没有任何区别，所以顶号流程与常规登录流程前期的执行是一摸一样的，只有在DbApp查询完这个Entity的数据的时候，会带上之前绑定的Proxy地址一起返回：

/**
 *  DBApp::GetEntityHandler override
 */
void LoginHandler::onGetEntityCompleted( bool isOK,
                    const EntityDBKey & entityKey,
                    const EntityMailBoxRef * pBaseEntityLocation )
{
	// 忽略一些错误处理代码

    entityKey_ = entityKey;

    if (pBaseEntityLocation != NULL)
    {
        baseRef_ = *pBaseEntityLocation;
        pBaseRef_ = &baseRef_;
    }
    else
    {
        pBaseRef_ = NULL;
    }

    if (pStrmDbID_)
    {
        // Means ekey.dbID was 0 when we called prepareCreateEntityBundle()
        // Now fix up everything.
        *pStrmDbID_ = entityKey.dbID;
    }

    this->checkOutEntity();
}

这个Proxy的地址就存储在pBaseEntityLocation里，当这个指针不是nullptr的时候，代表这个Entity对应的Proxy已经创建好了，baseRef就会被赋值为对应的Proxy的地址，随后在checkOutEntity的时候根据baseRef里是否有值来执行不同的操作：

/**
 *  This function checks out the login entity. Must be called after
 *  entity has been successfully retrieved from the database.
 */
void LoginHandler::checkOutEntity()
{
    if ((pBaseRef_ == NULL) &&
        DBApp::instance().onStartEntityCheckout( entityKey_ ))
    {
        // 之前通知baseappmgr创建proxy的逻辑 这里先忽略
    }
    else    // Checked out
    {
        DBApp::instance().onLogOnLoggedOnUser( entityKey_.typeID,
            entityKey_.dbID, pParams_, clientAddr_, replyAddr_, replyID_,
            pBaseRef_, dataForClient_, dataForBaseEntity_ );

        delete this;
    }
}

我们重点关注的是这里的else分支，即尝试登录到已登录对象onLogOnLoggedOnUser:

/**
 *  This method is called when there is a log on request for an entity that is
 *  already logged on.
 */
void DBApp::onLogOnLoggedOnUser( EntityTypeID typeID, DatabaseID dbID,
    LogOnParamsPtr pParams,
    const Mercury::Address & clientAddr, const Mercury::Address & replyAddr,
    Mercury::ReplyID replyID, const EntityMailBoxRef * pExistingBase,
    const BW::string & dataForClient, const BW::string & dataForBaseEntity )
{
    // 先忽略一些容错代码

    INFO_MSG( "DBApp::onLogOnLoggedOnUser: name = %s. databaseID = "
                "%" FMT_DBID ". typeID = %d. entityID = %d. BaseApp = %s\n",
                pParams->username().c_str(),
                dbID,
                typeID,
                pExistingBase->id,
                pExistingBase->addr.c_str() );

    // Log on to existing base
    Mercury::ChannelSender sender(
        DBApp::getChannel( pExistingBase->addr ) );

    Mercury::Bundle & bundle = sender.bundle();
    bundle.startRequest( BaseAppIntInterface::logOnAttempt,
        new RelogonAttemptHandler( pExistingBase->type(), dbID,
            replyAddr, replyID, pParams, clientAddr, dataForClient ) );

    bundle << pExistingBase->id;
    bundle << clientAddr;
    bundle << pParams->encryptionKey();
    bundle << dataForBaseEntity;
}

onLogOnLoggedOnUser在处理完一些错误情况之后，会将新客户端的信息打包到logOnAttempt这个请求中，并发送到现有的Proxy的通信地址pExistingBase->addr 上,然后将RelogonAttemptHandler注册为这个RPC的回包函数。

当BaseApp接收到这个RPC之后，先通过id找到对应的Proxy，然后调用脚本层的回调onLogOnAttempt来判定是否接受这个顶号操作，根据接受的结果来决定后续的顶号流程：

/**
 *	This method handles a message from the database telling us that a player is
 *	trying to log on to an active entity.
 */
void BaseApp::logOnAttempt( const Mercury::Address & srcAddr,
		const Mercury::UnpackedMessageHeader & header,
		BinaryIStream & data )
{
	MF_ASSERT( srcAddr == this->dbApp().addr() );

	EntityID id;
	Mercury::Address clientAddr;
	data >> id >> clientAddr;

	Base * pBase = bases_.findEntity( id );

	if (pBase == NULL)
	{
		WARNING_MSG( "BaseApp::logOnAttempt: No base %u\n", id );

		Mercury::ChannelSender sender( this->dbApp().channel() );
		sender.bundle().startReply( header.replyID );
		sender.bundle() << BaseAppIntInterface::LOG_ON_ATTEMPT_WAIT_FOR_DESTROY;
		return;
	}

	// We never expect this to happen, but reject the login just in case
	IF_NOT_MF_ASSERT_DEV( pBase->isProxy() )
	{
		ERROR_MSG( "BaseApp::logOnAttempt:"
					"%u is not a proxy, rejecting login attempt.\n",
				id );

		Mercury::ChannelSender sender( this->dbApp().channel() );
		sender.bundle().startReply( header.replyID );
		sender.bundle() << BaseAppIntInterface::LOG_ON_ATTEMPT_REJECTED;
		return;
	}

	Proxy * pProxy = static_cast< Proxy * >( pBase );
	PyObject * pResult;

	BW::string encryptionKey;
	data >> encryptionKey;

	BW::string logOnData;
	data >> logOnData;

	bool tookControl;
	PyObject* pFunction = PyObject_GetAttrString( pBase, "onLogOnAttempt" );
	if (pFunction)
	{
		// 暂时省略脚本层处理顶号判定的代码
	}
	else
	{
		NOTICE_MSG( "BaseApp::logOnAttempt: "
				"Rejecting relogon attempt for entity %u. "
				"No script method %s.onLogOnAttempt\n",
			id, pBase->pType()->name() );
		PyErr_Clear();
		tookControl = false;
	}

	// 判定了是否能被顶号的后处理
}

这里脚本层onLogOnAttempt的处理主要是处理一些奇怪的边界条件，例如脚本调用trace，以及当前entity正在被销毁。判定的结果会存储在tookControl这个变量里，代表是否允许顶号， 然后再根据这个变量来做后续的逻辑:

if (tookControl)
{
	if (!clientAddr.ip)
	{
		// only clear base's client channel if this is not a web login
		// (check clientAddr.ip)
		INFO_MSG( "BaseApp::logOnAttempt: "
					"For %u from web login.\n", pBase->id() );
	}
	else 
	{
		// 省略一些错误检查

		pProxy->logOffClient( /* shouldCondemnChannel */ true );
	}

	pProxy->completeReLogOnAttempt( clientAddr, header.replyID,
		encryptionKey );
}
else
{
	NOTICE_MSG( "BaseApp::logOnAttempt: "
					"Rejecting relogin attempt. " \
					"Have not taken control.\n" );

	Mercury::ChannelSender sender( this->dbApp().channel() );
	Mercury::Bundle & bundle = sender.bundle();
	bundle.startReply( header.replyID );

	bundle << BaseAppIntInterface::LOG_ON_ATTEMPT_REJECTED;
}

这里很神奇的就是，如果允许被顶号，则会立即执行logoffClient来关掉老的客户端连接。如果老的客户端连接存在，则在关闭之前会发送一个ClientInterface::loggedOffArgs的消息到客户端，以通知其下线理由为被顶号：

/**
 *  Send the client a disconnect message, then disconnect.
 *
 *	@param shouldCondemnChannel	True if any client channel should be condemned,
 *								false if it should be immediately destroyed.
 *								Generally true unless it was a client time-out
 *								or other situation where the client channel is
 *								probably no longer in a good state.
 */
void Proxy::logOffClient( bool shouldCondemnChannel )
{
	AUTO_SCOPED_THIS_ENTITY_PROFILE;

	// Abort any pending downloads, and then tell the script about it after
	// the client has gone.
	DownloadCallbacks callbacks;
	dataDownloads_.abortDownloads( callbacks );

	if (this->isClientConnected())
	{
		// Send a message to the client telling it that it has been logged off.
		// The reason parameter is not yet used.
		Mercury::Bundle & bundle = pClientChannel_->bundle();
		ClientInterface::loggedOffArgs::start( bundle ).reason = 0;
		this->sendBundleToClient();
	}

	this->detachFromClient( shouldCondemnChannel );

	callbacks.triggerCallbacks( this );
}

关闭客户端连接之后，再执行detachFromClient来清理掉之前的连接信息，注意这里的shouldCondemnChannel是true，所以这个通道会存在一个备份之中，等待所有数据下发完成，而不是之前超时的时候直接强制关闭。

detachFromClient之后通过completeReLogOnAttempt来通知DbApp顶号已完成了老连接的销毁，这里会通过prepareForLogin生成一个新的LoginKey,同时将这个(LoginKey,Proxy)添加到PendingLogin这个Map里：

/**
 *	This method completes a re-log-on attempt.
 *
 *	@param clientAddress 	The client address that is now expected to log on.
 *	@param replyID 			The reply ID for the request for onLogOnAttempt.
 *	@param encryptionKey	The encryption key to be used.
 */
void Proxy::completeReLogOnAttempt( const Mercury::Address & clientAddress,
		Mercury::ReplyID replyID,
		const BW::string & encryptionKey )
{
	INFO_MSG( "Proxy::completeReLogOnAttempt( %s %d ): "
			"Waiting to accept re-log-on attempt from %s\n",
		this->pType()->name(),
		id_,
		clientAddress.c_str() );

	BaseApp & baseApp = BaseApp::instance();

	SessionKey loginKey = 0;
	if (clientAddress.ip)
	{
		loginKey = this->prepareForLogin( clientAddress );
	}

	this->encryptionKey( encryptionKey );

	Mercury::ChannelSender sender( baseApp.dbApp().channel() );
	Mercury::Bundle & bundle = sender.bundle();
	bundle.startReply( replyID );

	bundle << BaseAppIntInterface::LOG_ON_ATTEMPT_TOOK_CONTROL;

	// This needs to match what the BaseAppMgr sends back to the
	// database.
	bundle << baseApp.extInterface().address();
	bundle << this->baseEntityMailBoxRef();
	bundle << loginKey;

	pPendingReLogOn_.reset( NULL );
}

当DbApp收到这个回包之后，之前创建的RelogonAttemptHandler就会被执行，这里会通知LoginApp当前登录成功，并附上现有Proxy的地址以及新的LoginKey：

/*
 *	Mercury::ReplyMessageHandler override.
 */
void RelogonAttemptHandler::handleMessage(
	const Mercury::Address & source,
	Mercury::UnpackedMessageHeader & header,
	BinaryIStream & data,
	void * arg )
{
	uint8 result;
	data >> result;

	if (hasAborted_)
	{
		DEBUG_MSG( "RelogonAttemptHandler: DBID %" FMT_DBID ": "
				"Ignoring BaseApp reply, re-logon attempt has been aborted.\n",
			ekey_.dbID );

		// Delete ourselves as we have been aborted.
		delete this;
		return;
	}

	switch (result)
	{
	case BaseAppIntInterface::LOG_ON_ATTEMPT_TOOK_CONTROL:
	{
		INFO_MSG( "RelogonAttemptHandler: DBID %" FMT_DBID ": "
				"It's taken over.\n",
			ekey_.dbID );
		Mercury::Address proxyAddr;
		data >> proxyAddr;

		EntityMailBoxRef baseRef;
		data >> baseRef;

		replyBundle_.startReply( replyID_ );

		// Assume success.
		replyBundle_ << (uint8)LogOnStatus::LOGGED_ON;
		replyBundle_ << proxyAddr;
		replyBundle_.transfer( data, data.remainingLength() );
		replyBundle_ << dataForClient_;

		DBApp::instance().interface().sendOnExistingChannel( replyAddr_,
				replyBundle_ );

		delete this;

		break;
	}
	// 省略其他错误情况
	default:
		CRITICAL_MSG( "RelogonAttemptHandler: DBID %" FMT_DBID ": "
				"Invalid result %d\n",
			ekey_.dbID,
			int(result) );
		delete this;
		break;
	}
}

对于LoginApp来说，顶号成功与登录成功的处理是一样的，所以这里不再介绍相关处理代码。客户端接收到登录成功的消息之后，再向Proxy发送一个正常的登录请求, 此时根据LoginKey去查找对应的PendingLogin，并获取对应的Proxy，并执行attachToClient来重新绑定客户端连接，这样完整的顶号流程就结束了。

/**
 *	This method is called by a client to make initial contact with the BaseApp.
 *	It should be called after the client has logged in via the LoginApp.
 */
void LoginHandler::login( Mercury::NetworkInterface & networkInterface,
			const Mercury::Address& srcAddr,
			const Mercury::UnpackedMessageHeader& header,
			const BaseAppExtInterface::baseAppLoginArgs & args )
{
	PendingLogins::iterator pendingIter = pPendingLogins_->find( args.key );

	if (pendingIter == pPendingLogins_->end())
	{
		INFO_MSG( "LoginHandler::login(%s): "
				"No pending login for loginKey %u. Attempt = %u\n",
			srcAddr.c_str(), args.key, args.numAttempts );
		// Bad bundle so break out of dispatching the rest.
		header.breakBundleLoop();

		return;
	}

	const PendingLogin & pending = pendingIter->second;

	SmartPointer<Proxy> pProxy = pending.pProxy();

	// 省略一些代码

	this->updateStatistics( srcAddr, pending.addrFromLoginApp(), args.numAttempts );

	pPendingLogins_->erase( pendingIter );

	if (pProxy->attachToClient( srcAddr, header.replyID,
			header.pChannel.get() ))
	{
		INFO_MSG( "LoginHandler::login: "
			"%u attached from %s. Attempt %u\n",
			pProxy->id(), srcAddr.c_str(), args.numAttempts );
	}
}

现在再来思考一下在当前的顶号流程中是否会出现与mosaic_game顶号过程中类似的老客户端数据发往新客户端的行为。仔细想想，还真可能有，即在CellApp上的RealEntity发往老客户端的数据在顶号结束绑定新客户端之后才被Proxy收到，此时如果直接往新的ClientChannel进行投递的话，就会有出现上面担心的问题。这个问题BigWorld中也考虑到了，他这里引入了一个切换客户端之后通知RealEntity的握手机制，具体细节在下面的小节里进行阐述。

客户端进入场景流程

前面的四个小节里我们详尽的介绍了客户端的登录、断线、顶号流程，了解了为了有效的维护Proxy与客户端之间的消息通信所做的各项准备工作。但是消息通道的建立只是开始，我们还需要通过这个通道来同步角色在服务器的各种状态到客户端，然而此时客户端并没有任何entity对象。因此客户端知道Proxy已经接纳了当前的登录之后，会主动发送一个enableEntities请求过来，通知服务器可以向客户端推送相关的entity数据了：

/**
 *	This method handles a request from the client to enable or disable updates
 *	from the cell. It forwards this message on to the cell.
 */
void Proxy::enableEntities()
{
	DEBUG_MSG( "Proxy::enableEntities(%u)\n", id_ );

	// if this is the first we've heard of it, then send the client the props
	// it shares with us, call the base script...
	if (!basePlayerCreatedOnClient_)
	{
		this->addCreateBasePlayerToChannelBundle();
		this->sendExposedForReplayClientPropertiesToCell();

		if (pBufferedClientBundle_.get())
		{
			// Make sure the BasePlayer arrives before the buffered messages
			this->sendBundleToClient();

			// TODO: This is because Channel::send will not send arbitrary
			// bundles on a channel with a bundle primer present.
			// Would it work to copy the messages into the clientBundle instead?
			pClientChannel_->bundlePrimer( NULL );

			Mercury::Bundle & bundle = pClientChannel_->bundle();
			pBufferedClientBundle_->applyToBundle( bundle );
			pClientChannel_->send( &bundle );
			pClientChannel_->bundlePrimer( &clientBundlePrimer_ );
			pBufferedClientBundle_.reset();
		}
	}

	// ... and tell the cell the game is on
	if (!entitiesEnabled_)
	{
		entitiesEnabled_ = true;

		if (this->hasCellEntity())
		{
			this->sendEnableDisableWitness( /*enable:*/true );

			// remove ProxyPusher
			if (pProxyPusher_ != NULL)
			{
				delete pProxyPusher_;
				pProxyPusher_ = NULL;
			}
		}
		else
		{
			// Add a proxy pusher because we don't have a cell to do it.
			if (pProxyPusher_ == NULL)
			{
				pProxyPusher_ = new ProxyPusher( this );
			}
		}
	}

	if (shouldRunCallbackOnEntitiesEnabled_)
	{
		// call the script and let it have its naughty way with the client
		Script::call( PyObject_GetAttrString( this, "onEntitiesEnabled" ),
			PyTuple_New( 0 ), "", true );

		shouldRunCallbackOnEntitiesEnabled_ = false;
	}
}

这个函数分为两个部分，一个是通过addCreateBasePlayerToChannelBundle将当前Player Entity的数据通过ClientInterface::createBasePlayer请求下发到客户端，让客户端创建PlayerEntity。这些数据在创建Proxy的时候就已经被DBApp填充好了，设置在当前Proxy的Properties里，打包的时候使用EntityDescription::FROM_BASE_TO_CLIENT_DATA来过滤出所有客户端所需要的属性，并组成一个PythonDict，添加到bundle里：

/**
 *	This method adds the createBasePlayer message to the given bundle
 *
 *	It should immediately follow a successful login or full
 *	entity reset, so the client is never operating without
 *	a Base Player entity.
 *	Note: When this method is called,
 *		Proxy::sendExposedForReplayClientPropertiesToCell() should be called
 *		together at the same time.
 *
 *	@param bundle	The Mercury::Bundle to add the message to
 */
void Proxy::addCreateBasePlayerToChannelBundle()
{
	DEBUG_MSG( "Proxy::addCreateBasePlayerToChannelBundle(%u): "
			"Creating player on client\n",
		id_ );

	MF_ASSERT( pClientChannel_ != NULL );
	MF_ASSERT( shouldRunCallbackOnEntitiesEnabled_ == false );
	MF_ASSERT( basePlayerCreatedOnClient_ == false );

	Mercury::Bundle & bundle = pClientChannel_->bundle();

	bundle.startMessage( ClientInterface::createBasePlayer );
	bundle << id_ << pType_->description().clientIndex();
	this->addAttributesToStream( bundle,
		EntityDescription::FROM_BASE_TO_CLIENT_DATA );

	shouldRunCallbackOnEntitiesEnabled_ = true;
	basePlayerCreatedOnClient_ = true;
}

/**
 *	This method writes attributes of the entity related to the given dataDomains
 *	to stream.
 */
bool Base::addAttributesToStream( BinaryOStream & stream, int dataDomains )
{
	const EntityDescription & entityDesc = this->pType()->description();
	ScriptObject self( this, ScriptObject::FROM_BORROWED_REFERENCE );
	ScriptDict attrs = createDictWithAllProperties(entityDesc, 
		self, pEntityDelegate_.get(), dataDomains);
	
	if (!attrs)
	{
		return false;
	}
	return entityDesc.addDictionaryToStream( attrs, stream, dataDomains );
}

另外一个部分是通过sendEnableDisableWitness来通知对应的RealEntity(如果已经创建了RealEntity)，新的客户端已经允许同步AOI内的其他对象，可以往客户端发送所有AOI可见对象的相关数据了：

/**
 *	This method sends an enable or disable witness message to our cell entity.
 *
 *	@param enable		whether to enable or disable the witness
 *	@param isRestore 	is this an explicit witness enable/disable send as a
 *						result of a restore cell entity?
 */
void Proxy::sendEnableDisableWitness( bool enable, bool isRestore )
{
	AUTO_SCOPED_THIS_ENTITY_PROFILE;

	Mercury::Bundle & bundle = this->cellBundle();
	bundle.startRequest( CellAppInterface::enableWitness,
			new EnableWitnessReplyHandler( this ) );

	bundle << id_;
	bundle << isRestore;

	++numOutstandingEnableWitness_;
	cellHasWitness_ = enable;

	if (enable)
	{
		bundle << BaseAppConfig::bytesPerPacketToClient();
	}
	// else just send an empty stream

	this->sendToCell();	// send it straight away
}

在之前的detachFromClient函数中，也会调用sendEnableDisableWitness，不过此时的参数是false，这样就会通知RealEntity先暂停AOI的同步，来节省流量和CPU等资源。因为即使发过来，也会因为没有客户端而被抛弃。

注意到sendEnableDisableWitness这里有一个特殊的自增操作，++numOutstandingEnableWitness_，这个就是记录现在还有多少个CellAppInterface::enableWitness没有收到reply。如果收到了reply, 则在EnableWitnessReplyHandler会对这个字段做减一操作：

/**
 *	This reply handler is used to keep track of how many outstanding
 *	enableWitness there are for each proxy.
 */
class EnableWitnessReplyHandler :
	public Mercury::ShutdownSafeReplyMessageHandler
{
public:
	EnableWitnessReplyHandler( ProxyPtr pProxy ) :
		pProxy_( pProxy )
	{
	}

	void handleMessage( const Mercury::Address& /*srcAddr*/,
			Mercury::UnpackedMessageHeader& /*header*/,
			BinaryIStream& data, void * /*arg*/ )
	{
		this->onReply();
	}

	void onReply()
	{
		pProxy_->onEnableWitnessAck();
		delete this;
	}

private:
	ProxyPtr pProxy_;
};
/**
 *	This method is called when there is confirmation that the witness has been
 *	created.
 */
void Proxy::onEnableWitnessAck()
{
	MF_ASSERT( numOutstandingEnableWitness_ > 0 );
	--numOutstandingEnableWitness_;
}

如果numOutstandingEnableWitness_的值不为0，则代表CellApp中的RealEntity暂时还没有确认我们发出的客户端连接状态变化。此时可以认为RealEntity下发的客户端数据的目的地是老客户端，而不是新客户端，因此应该抛弃。所以只要在所有往客户端投递数据的接口里做一下这个握手检查，就能有效的避免顶号之后目的地为老客户端的数据被传递到新客户端的错误：

	bool Proxy::hasOutstandingEnableWitness() const
		{ return numOutstandingEnableWitness_ != 0; }
/**
 *	This message is the cell telling us that it has now sent us all the
 *	updates for the given tick, and we should forward them on to the client.
 */
void Proxy::sendToClient()
{
	// Do nothing. It's for an old client.
	if (!this->hasOutstandingEnableWitness())
	{
		this->sendBundleToClient();
	}
}

/**
 *	This method forwards this message to the client.
 */
void Proxy::sendMessageToClientHelper( BinaryIStream & data, bool isReliable )
{
	AUTO_SCOPED_THIS_ENTITY_PROFILE;

	if (this->hasOutstandingEnableWitness())
	{
		// Do nothing. It's for an old client.
		data.finish();
		return;
	}
	// 省略很多代码
}

/**
 *	This method forwards this message to the client (reliably)
 */
#define STRUCT_CLIENT_MESSAGE_FORWARDER( MESSAGE )							\
void Proxy::MESSAGE( const BaseAppIntInterface::MESSAGE##Args & args )		\
{																			\
	if (this->hasOutstandingEnableWitness())								\
	{																		\
		/* Do nothing. It's for an old client. */							\
	}																		\
	\\ 省略很多代码
}																			\

/**
 *	This method tells us about a change in the status of one of our wards.
 *	We modify our internal list, then forward the message on to the client.
 */
void Proxy::modWard( const BaseAppIntInterface::modWardArgs & args )
{
	if (this->hasOutstandingEnableWitness())
	{
		// TODO: Should make sure that the wards are reset when given a new
		// client.
		ERROR_MSG( "Proxy::modWard( %d ): Has outstanding enableWitness\n",
						id_ );
		return;
	}
	// 省略很多代码
}

// 省略其他往客户端投递数据的接口

当客户端的PlayerEntity创建完成之后，就可以准备进入特定的游戏场景了。这个部分的逻辑并没有写在cpp代码中，而是在Base对象上增加了几个可以给Python脚本调用的接口，让逻辑层自己决定进入场景的相关参数，典型接口就是createCellEntity:

	/*~ function Base.createCellEntity
	 *	@components{ base }
	 *	<i>createCellEntity</i> makes a request to create an associated entity
	 *	within a cell.
	 *
	 *	The information used to create the cell entity is stored in the
	 *	cellData property of this entity. This property is a dictionary
	 *	corresponding to the values in the entity's .def file plus a
	 *	"position" and "direction" entries for the entity's position
	 *	and (roll, pitch, yaw) direction as well as optional "spaceID"
	 *	and "templateID" entries. In case if "templateID" entry is present
	 *	the non-spatial properties are not transmitted, instead the "templateID"
	 *	is used on CellApp to populate the cell entity's properties 
	 *	from a local storage.
	 *
	 *	If nearbyMB is not passed in, the "spaceID" entry in cellData is
	 *	used to indicate which space to create the cell entity in.
	 *
	 *	@param nearbyMB an optional mailbox argument which is used to indicate
	 *	which space to create the cell entity in. Ideally, the two entities
	 *	are near so that it is likely that the correct cell will be found
	 *	immediately. Either the base or the cell mailbox of the nearby entity
	 *	can be used; when using base or cell-via-base mailboxes, the entity
	 *	reference will be passed to the __init__() method of the cell entity so
	 *	the nearby entity's position can be used to set the new cell entity's
	 *	position.
	 */
	PY_METHOD( createCellEntity )

类似的脚本接口还有createInDefaultSpace, 其实这两个接口都是下面接口的封装：

/**
 *	This method creates the cell entity associated with this entity into the
 *	input space.
 */
bool Base::createInSpace( SpaceID spaceID, const char * pyErrorPrefix )
{
	BaseApp & app = BaseApp::instance();

	// TODO:
	// As an optimisation, try to find a cell entity mailbox for an existing
	// base entity that is in the same space.
	//
	// This is currently not implemented as there is a potential race-condition.
	// The entity may currently be in the same space but may be in a different
	// space by the time the createCellEntity message arrives.

	std::auto_ptr< Mercury::ReplyMessageHandler > pHandler(
		this->prepareForCellCreate( pyErrorPrefix ) );

	if (!pHandler.get())
	{
		return false;
	}

	Mercury::Channel & channel = BaseApp::getChannel( app.cellAppMgrAddr() );
	// We don't use the channel's own bundle here because the streaming might
	// fail and the message might need to be aborted halfway through.
	std::auto_ptr< Mercury::Bundle > pBundle( channel.newBundle() );
	pBundle->startRequest( CellAppMgrInterface::createEntity, pHandler.get() );
	*pBundle << spaceID;

	// stream on the entity channel version
	*pBundle << this->channel().version();

	*pBundle << false; /* isRestore */

	// See if we can add the necessary data to the bundle
	if (!this->addCellCreationData( *pBundle, pyErrorPrefix ))
	{
		isCreateCellPending_ = false;
		isGetCellPending_ = false;

		return false;
	}

	channel.send( pBundle.get() );
	pHandler.release(); // Handler deletes itself on callback.

	return true;
}

这个函数负责往CellAppMgr发起一个createEntity的请求，参数里传入了要进入的场景标识符spaceID，同时使用addCellCreationData将当前Base里存储的创建CellEntity的相关信息也打包Bundle中。这个addCellCreationData实现比较长，这里就大概介绍一下需要打包那些数据：

1. 进入场景的位置、朝向、是否贴地
2. 当前Base所在的BaseApp的通信地址
3. 当前Base属性系统所采取的模板ID,属性模板负责填充一些属性的默认值，这样就没必要将默认值传递过去，只需要传递模板ID
4. 当前Base的属性系统里标记为exposedForReplay 的client-server属性。

数据打包完成之后，就会发送到CellAppMgr来处理，此时会根据传入的spaceID来找到对应的Space对象，然后执行在Space内创建RealEntity的操作：

/**
 *	This method creates a new entity on the system. It finds the appropriate
 *	cell to create the entity on and does so.
 *
 *	@todo Currently adds the entity to the first space but the correct space
 *			will need to be specified in the message.
 */
void CellAppMgr::createEntity( const Mercury::Address& srcAddr,
		const Mercury::UnpackedMessageHeader& header,
		BinaryIStream & data )
{
	SpaceID spaceID;
	data >> spaceID;
	Space * pSpace = this->findSpace( spaceID );

	if (pSpace == NULL)
	{
		ERROR_MSG( "CellAppMgr::createEntity: Invalid space id %u\n", spaceID );
		// Rely on createEntityCommon to send the error reply.
	}
	else
	{
		pSpace->hasHadEntities( true );
	}

	this->createEntityCommon( pSpace, srcAddr, header, data );
}

这个createEntityCommon负责根据出生点的位置来寻找一个合适的Cell，并将这个createEntity的请求转发到Cell上：

/**
 *	This private method is used by createEntity and createSpace to implement
 *	their common functionality.
 */
void CellAppMgr::createEntityCommon( Space * pSpace,
		const Mercury::Address& srcAddr,
		const Mercury::UnpackedMessageHeader& header,
		BinaryIStream & data )
{
	Mercury::ChannelVersion channelVersion = Mercury::SEQ_NULL;
	data >> channelVersion;
	bool isRestore;
	data >> isRestore;

	StreamHelper::AddEntityData entityData;
	StreamHelper::removeEntity( data, entityData );

	const Vector3 & pos = entityData.position;
	CellData * pCellData = pSpace ? pSpace->findCell( pos.x, pos.z ) : NULL;

	if (pCellData)
	{
		Mercury::Bundle & bundle = pCellData->cellApp().bundle();

		bundle.startRequest( CellAppInterface::createEntity,
			new CreateEntityReplyHandler( srcAddr, header.replyID ) );

		bundle << pCellData->space().id();
		bundle << channelVersion;

		bundle << isRestore;
		StreamHelper::addEntity( bundle, entityData );

		bundle.transfer( data, data.remainingLength() );

		pCellData->cellApp().send();
	}
	else
	{
		ERROR_MSG( "CellAppMgr::createEntity: "
					"No cell found to place entity\n" );
		data.finish();

		Mercury::ChannelSender sender( CellAppMgr::getChannel( srcAddr ) );
		Mercury::Bundle & bundle = sender.bundle();

		bundle.startReply( header.replyID );
		bundle << NULL_ENTITY_ID;
	}
}

当这个Cell接收到CreateEntity请求之后，就开始使用传入的相关参数真正的创建一个Entity，并将创建的Entity的唯一标识符作为结果返回给CellAppMgr:

/**
 *	This method creates a new real entity on this cell according to the
 *	parameters in 'data'.
 *
 *	@param srcAddr	The address from which this request originated.
 *	@param header	The mercury header
 *	@param data		The data stream
 *	@param pNearbyEntity A pointer to a nearby entity to use during entity
 *						creation.
 *
 *	@see createEntityInternal
 */
void Cell::createEntity( const Mercury::Address& srcAddr,
		const Mercury::UnpackedMessageHeader& header,
		BinaryIStream & data,
		EntityPtr pNearbyEntity )
{
	Mercury::ChannelVersion channelVersion = Mercury::SEQ_NULL;
	data >> channelVersion;

	bool isRestore;
	data >> isRestore;

	EntityPtr pEntity = this->createEntityInternal( data, ScriptDict(),
		isRestore, channelVersion, pNearbyEntity );

	if (isRestore)
	{
		return;
	}

	if (header.replyID == Mercury::REPLY_ID_NONE)
	{
		WARNING_MSG( "Cell::createEntity: Handling non-request createEntity\n" );
		return;
	}

	if (pNearbyEntity == NULL)
	{
		// Only CellAppMgr sends without a nearby entity.
		CellAppMgrGateway & cellAppMgr = CellApp::instance().cellAppMgr();

		cellAppMgr.bundle().startReply( header.replyID );
		cellAppMgr.bundle() << (pEntity ? pEntity->id() : NULL_ENTITY_ID);
		cellAppMgr.send();
		return;
	}

	// 忽略一些代码
}

当createEntityInternal内一个Entity被创建的时候，对应的RealEntity就会在Entity::initReal中尝试被创建,并通过addRealEntity加入到当前的Cell中：

// Build up the Entity structure
EntityPtr pNewEntity = space_.newEntity( id, entityTypeID );

if (!pNewEntity)
{
	return NULL;
}

MF_ASSERT( pNewEntity->nextInChunk() == NULL );
MF_ASSERT( pNewEntity->prevInChunk() == NULL );
MF_ASSERT( pNewEntity->pChunk() == NULL );

Entity::callbacksPermitted( false ); // {

if (!pNewEntity->initReal( data, properties, isRestore, channelVersion,
			pNearbyEntity ))
{
	// 忽略容错代码

	return NULL;
}
// And add it to our list of reals
// TODO: If init method destroyed or teleported this entity
// then we should not do this (and destroying/offloading
// would have caused an assert when it removed the entity anyway)
//
// Delay sending the currentCell message to the base so that it will be on
// the same bundle as the backup. This guarantees that there is no window
// between losing cellData and getting the first backup data.
this->addRealEntity( pNewEntity.get(), /*shouldSendNow:*/false );

INFO_MSG( "Cell::createEntityInternal: %s %-10s (%u)\n",
		isRestore ? "Restored" : "New",
		pNewEntity->pType() ?
			pNewEntity->pType()->name() : "INVALID",
		pNewEntity->id() );

Entity::population().notifyObservers( *pNewEntity );
// Note: This causes the currentCell message from above to be sent too.
if (pNewEntity->pReal())
{
	pNewEntity->pReal()->backup();	// Send backup to BaseApp immediately.
}

return pNewEntity;

对于Space里的一个Entity，要么其拥有RealEntity，要么其拥有GhostEntity，所以这里的注释标注了initReal与initGhost两者必须调用其中一个，且必须在Entity构造之后立即调用：

/**
 *	This method should be called on a newly created entity to make it a real
 *	entity. Either this method or initGhost should be called immediately after
 *	the constructor.
 *
 *	@see initGhost
 */
bool Entity::initReal( BinaryIStream & data, const ScriptDict & properties,
		bool isRestore,
		Mercury::ChannelVersion channelVersion,
		EntityPtr pNearbyEntity )

由于我们当前是第一次进入Space，所以创建的肯定是RealEntity。

当CellAppMgr的CreateEntityReplyHandler接收到创建结果之后，其唯一作用就是将这个回复进一步的转发到发起createEntity调用的Base上的CreateCellEntityHandler去处理：

/*
 *	This method handles the reply from the cellapp.
 */
void CreateCellEntityHandler::handleMessage( const Mercury::Address & source,
		Mercury::UnpackedMessageHeader & header,
		BinaryIStream & data, void * arg )
{
	EntityID entityID;
	data >> entityID;

	// MF_ASSERT( entityID == pBase_->id() );
	if (entityID == pBase_->id())
	{
		// INFO_MSG( "CreateCellEntityHandler::handleMessage: "
		// 	"Cell entity (%lu) created\n",
		// 	entityID );
		pBase_->cellCreationResult( true );
	}
	else
	{
		WARNING_MSG( "CreateCellEntityHandler::handleMessage: "
			"Failed to create associated cell entity for %u.\n"
			"\tResponse was entity id %u\n", pBase_->id(), entityID );
		pBase_->cellCreationResult( false );
	}

	delete this;
}

当成功创建的时候，返回的EntityId肯定是当前Base存储的EntityId，成功之后会执行cellCreationResult来标记当前已经在Cell里创建好了RealEntity:

/**
 *	This method is called when we find out whether or not creating an
 *	entity on the cell succeeded.
 */
void Base::cellCreationResult( bool success )
{
	if (isDestroyed_)
	{
		return;
	}

	// isCreateCellPending_ is also clear before onGetCell is called
	if (!isCreateCellPending_)
	{
		// This may occur in a very rare situation. If the cell was created via
		// the CellAppMgr, it's possible (although unlikely) for the CellApp to
		// have responded with success but the CellAppMgr responds with failure.
		// This occurrs when the CellApp crashes while a 
		// CreateEntityReplyHandler is outstanding but after setCurrentCell.

		if (!success)
		{
			ERROR_MSG( "Base::cellCreationResult: Ignoring failure after "
					   "setCurrentCell\n" );
		}
		return;
	}

	isCreateCellPending_ = false;

	if (!success)
	{
		isGetCellPending_ = false;
		Script::call( PyObject_GetAttrString( this, "onCreateCellFailure" ),
				PyTuple_New( 0 ), "onCreateCellFailure", true );
		DEBUG_MSG( "Base::cellCreationResult: Failed for %u\n", id_ );
	}
}

但是只是标记好了这个isCreateCellPending_，现在的Base里还是不知道所创建的RealEntity的相关信息。这些信息通过addRealEntity触发填充和发送，调用者是createEntityInternal，大家可以回顾一下:

/**
 *	This method adds the input entity to the cell's internal list of real
 *	entities.
 *
 *	It is called from Cell::createEntity and by the Entity itself when it is
 *  onloaded.
 */
void Cell::addRealEntity( Entity * pEntity, bool shouldSendNow )
{
	if (!pEntity->isReal())
	{
		ERROR_MSG( "Cell::addRealEntity called on ghost entity id %u!\n",
			pEntity->id() );
		return;
	}

	pEntity->informBaseOfAddress( CellApp::instance().interface().address(),
		this->spaceID(), shouldSendNow );

	realEntities_.add( pEntity );
}

informBaseOfAddress负责将当前的CellApp的地址与当前场景的SpaceID塞入到BaseAppIntInterface::currentCellArgs这里：

/**
 *	This method informs the base entity of the address of the cell entity.
 */
void Entity::informBaseOfAddress( const Mercury::Address & addr,
		SpaceID spaceID, bool shouldSendNow )
{
	AUTO_SCOPED_THIS_ENTITY_PROFILE;

	// TODO: Move this to RealEntity
	MF_ASSERT( this->isReal() );

	if (this->hasBase())
	{
		Mercury::Bundle & bundle = pReal_->channel().bundle();

		BaseAppIntInterface::setClientArgs & setClientArgs =
			BaseAppIntInterface::setClientArgs::start( bundle );
		setClientArgs.id = id_;

		// Our base knowing where we are is considered to be critical.  In
		// particular, if the base doesn't know about this real it may try to
		// restore this entity somewhere else following a cellapp crash which
		// will cause the !pOtherGhost->isReal() assertion.
		BaseAppIntInterface::currentCellArgs & currentCellArgs =
			BaseAppIntInterface::currentCellArgs::start(
				bundle, Mercury::RELIABLE_CRITICAL );

		currentCellArgs.newSpaceID = spaceID;
		currentCellArgs.newCellAddr = addr;

		if (shouldSendNow)
		{
			pReal_->channel().send();
		}
	}
}

当Base接收到currentCell这个RPC的时候，就会设置

/**
 *	This method is used to inform the base that the cell we send to has changed.
 */
void Base::currentCell( const Mercury::Address & srcAddr,
			const Mercury::UnpackedMessageHeader & header,
			const BaseAppIntInterface::currentCellArgs & args )
{
	this->setCurrentCell( args.newSpaceID, args.newCellAddr,
			&srcAddr );
}

/**
 *	This method is used to inform the base that the cell we send to has changed.
 */
void Base::setCurrentCell( SpaceID spaceID,
	const Mercury::Address & cellAppAddr,
	const Mercury::Address * pSrcAddr, bool shouldReset )
{
	// Make sure that we are still around after any script call.
	PyObjectPtr pThis = this;

	// If we're losing our cell entity, flush the channel.
	if (cellAppAddr == Mercury::Address::NONE)
	{
		if (pChannel_->isEstablished())
		{
			pChannel_->send();

			// We have to reset the channel here because we might get another
			// cell entity later on and that cell entity will expect the channel
			// to be in a reset state.  This will put the channel into the
			// 'wantsFirstPacket_' state, so even if a packet arrives from the
			// old app, it will be dropped.
			pChannel_->reset( Mercury::Address::NONE, false );
		}
	}

	// If we're getting a cell entity or offloading, just switch the address.
	else
	{
		// Usually we don't need to manually switch address here, since the
		// channel has the autoSwitchToSrcAddr flag enabled and it has already
		// been done by Mercury.  We still need to do this when called from
		// emergencySetCurrentCell() however.
		if (shouldReset)
		{
			pChannel_->reset( cellAppAddr );
		}
		else
		{
			pChannel_->setAddress( cellAppAddr );
		}
	}
	// 暂时先省略一部分代码
}

这个setCurrentCell的作用就是将pChannel_的地址设置为当前CellAppAddr，之后Base与RealEntity之间的通信就可以使用这个pChannel_了。同时我们再回顾一下Proxy与客户端的通信使用的是pClientChannel_，不要跟这里的pChannel_搞混了。

在pChannel_设置之后，setCurrentCell开始通知其他组件当前的RealEntity已经创建，下面就是之前省略的后续代码：

if (pCellEntityMailBox_ != NULL)
{
	bool hadCell = (pCellEntityMailBox_->address().ip != 0);
	bool haveCell = (cellAppAddr.ip != 0);

	pCellEntityMailBox_->address( pChannel_->addr() );
	spaceID_ = spaceID;

	if (hadCell != haveCell)
	{
		isGetCellPending_ = false;

		if (haveCell)
		{
			pCellData_ = NULL;
		}
		else
		{
			cellBackupData_.clear();
		}

		// inform the proxy that the cell entity has gone
		// (even if a new one is requested by the script method below)
		if (this->isProxy() && hadCell)
		{
			((Proxy*)this)->cellEntityDestroyed( pSrcAddr );
		}

		if (haveCell)
		{
			isCreateCellPending_ = false;
			// There might still be stuff waiting for a valid IP address to
			// come along
			pChannel_->send();
		}

		// call the script method notifying it of this event
		char * methodName = (char*)(haveCell ? "onGetCell" : "onLoseCell");
		PyObject * pMethod = PyObject_GetAttrString( this, methodName );

		if (pMethod == NULL)
			PyErr_Clear();

		// inform the proxy that the cell entity is ready
		if (this->isProxy() && this->hasCellEntity())
		{
			((Proxy*)this)->cellEntityCreated();
		}

		if (pMethod)
		{
			Script::call( pMethod, PyTuple_New( 0 ), methodName );
		}
		
		// notify the delegate about this event
		if (pEntityDelegate_)
		{
			if (haveCell)
			{
				//TODO: maybe we should add the handling of this condition to the delegate, 
				//      but meanwhile it's not needed/handled.
				//
				//pEntityDelegate_->onGetCell();
			}
			else
			{
				pEntityDelegate_->onLoseCell();
			}
		}
	}
}

由于这个SetCurrentCell应对了很多种RealEntity地址修改的情况，我们目前先聚焦于RealEntity第一次创建时的处理，也就是这里的cellEntityCreated函数：

/**
 *	This method deals with our cell entity being created.
 */
void Proxy::cellEntityCreated()
{
	AUTO_SCOPED_THIS_ENTITY_PROFILE;

	if (!entitiesEnabled_) return;
	MF_ASSERT( this->hasClient() );

	MF_ASSERT( this->hasCellEntity() );

	//  create the witness
	this->sendEnableDisableWitness( /*enable:*/true );

	// get rid of the proxy pusher now that the witness will be sending us
	// regular updates (the self motivator should definitely be there).
	MF_ASSERT( pProxyPusher_ != NULL );
	delete pProxyPusher_;
	pProxyPusher_ = NULL;
}

这里的sendEnableDisableWitness我们之前在顶号的时候介绍过了，其作用是通知RealEntity开始处理自身以及AOI向客户端的同步。

Unreal Engine 的玩家流程管理

端口监听

在UE中网络相关功能由网络驱动类型UNetDriver来提供，这个UNetDriver是一个虚基类，提供了各种网络收发和管理功能。实际被使用的主要是其子类IpNetDriver或者DemoNetDriver， IpNetDriver负责处理正常的游戏，而DemoNetDriver负责处理游戏录像的回放。UNetDriver可以继续扩展，但是每个扩展出来的子类都需要在BaseEngine.ini通过NetDriverDefinitions注册到引擎:

+NetDriverDefinitions=(DefName="GameNetDriver",DriverClassName="/Script/OnlineSubsystemUtils.IpNetDriver",DriverClassNameFallback="/Script/OnlineSubsystemUtils.IpNetDriver")
+NetDriverDefinitions=(DefName="BeaconNetDriver",DriverClassName="/Script/OnlineSubsystemUtils.IpNetDriver",DriverClassNameFallback="/Script/OnlineSubsystemUtils.IpNetDriver")
+NetDriverDefinitions=(DefName="DemoNetDriver",DriverClassName="/Script/Engine.DemoNetDriver",DriverClassNameFallback="/Script/Engine.DemoNetDriver")

同时引擎里提供一个CreateNamedNetDriver的接口，来通过传入的DefName来创建对应的NetDriver:

bool UEngine::CreateNamedNetDriver(UWorld *InWorld, FName NetDriverName, FName NetDriverDefinition)
{
	return UE::Private::CreateNamedNetDriver_Local(this, GetWorldContextFromWorldChecked(InWorld), NetDriverName, NetDriverDefinition);
}

UNetDriver* CreateNetDriver_Local(UEngine* Engine, FWorldContext& Context, FName NetDriverDefinition, FName InNetDriverName)
{
	UNetDriver* ReturnVal = nullptr;
	FNetDriverDefinition* Definition = nullptr;
	auto FindNetDriverDefPred =
		[NetDriverDefinition](const FNetDriverDefinition& CurDef)
	{
		return CurDef.DefName == NetDriverDefinition;
	};


	{
		Definition = Engine->NetDriverDefinitions.FindByPredicate(FindNetDriverDefPred);
	}

	if (Definition != nullptr)
	{
		UClass* NetDriverClass = StaticLoadClass(UNetDriver::StaticClass(), nullptr, *Definition->DriverClassName.ToString(), nullptr,
			LOAD_Quiet);

		// if it fails, then fall back to standard fallback
		if (NetDriverClass == nullptr || !NetDriverClass->GetDefaultObject<UNetDriver>()->IsAvailable())
		{
			NetDriverClass = StaticLoadClass(UNetDriver::StaticClass(), nullptr, *Definition->DriverClassNameFallback.ToString(),
				nullptr, LOAD_None);
		}

		if (NetDriverClass != nullptr)
		{
			ReturnVal = NewObject<UNetDriver>(GetTransientPackage(), NetDriverClass);

			check(ReturnVal != nullptr);

			const FName DriverName = InNetDriverName.IsNone() ? ReturnVal->GetFName() : InNetDriverName;
			const bool bInitializeWithIris = Engine->WillNetDriverUseIris(Context, NetDriverDefinition, DriverName);

			ReturnVal->SetNetDriverName(DriverName);
			ReturnVal->SetNetDriverDefinition(NetDriverDefinition);
			ReturnVal->PostCreation(bInitializeWithIris);

			new(Context.ActiveNetDrivers) FNamedNetDriver(ReturnVal, Definition);

			FWorldDelegates::OnNetDriverCreated.Broadcast(Context.World(), ReturnVal);
		}
	}


	if (ReturnVal == nullptr)
	{
		UE_LOG(LogNet, Log, TEXT("CreateNamedNetDriver failed to create driver from definition %s"), *NetDriverDefinition.ToString());
	}

	return ReturnVal;
}

当服务器启动并加载地图之后，就会去调用UWorld::Listen来开启网络的监听，也就是在这里会创建GameNetDriver:

bool UWorld::Listen( FURL& InURL )
{
#if WITH_SERVER_CODE
	LLM_SCOPE(ELLMTag::Networking);

	if( NetDriver )
	{
		GEngine->BroadcastNetworkFailure(this, NetDriver, ENetworkFailure::NetDriverAlreadyExists);
		return false;
	}

	// Create net driver.
	if (GEngine->CreateNamedNetDriver(this, NAME_GameNetDriver, NAME_GameNetDriver))
	{
		NetDriver = GEngine->FindNamedNetDriver(this, NAME_GameNetDriver);
		NetDriver->SetWorld(this);
		FLevelCollection* const SourceCollection = FindCollectionByType(ELevelCollectionType::DynamicSourceLevels);
		if (SourceCollection)
		{
			SourceCollection->SetNetDriver(NetDriver);
		}
		FLevelCollection* const StaticCollection = FindCollectionByType(ELevelCollectionType::StaticLevels);
		if (StaticCollection)
		{
			StaticCollection->SetNetDriver(NetDriver);
		}
	}

	if (NetDriver == nullptr)
	{
		GEngine->BroadcastNetworkFailure(this, NULL, ENetworkFailure::NetDriverCreateFailure);
		return false;
	}

	AWorldSettings* WorldSettings = GetWorldSettings();
	const bool bReuseAddressAndPort = WorldSettings ? WorldSettings->bReuseAddressAndPort : false;

	FString Error;
	if( !NetDriver->InitListen( this, InURL, bReuseAddressAndPort, Error ) )
	{
		// 忽略一些错误处理代码
		return false;
	}
	static const bool bLanPlay = FParse::Param(FCommandLine::Get(),TEXT("lanplay"));
	const bool bLanSpeed = bLanPlay || InURL.HasOption(TEXT("LAN"));
	if ( !bLanSpeed && (NetDriver->MaxInternetClientRate < NetDriver->MaxClientRate) && (NetDriver->MaxInternetClientRate > 2500) )
	{
		NetDriver->MaxClientRate = NetDriver->MaxInternetClientRate;
	}

	NextSwitchCountdown = NetDriver->ServerTravelPause;
	return true;
#else
	return false;
#endif // WITH_SERVER_CODE
}

当一个NetDriver被成功的创建之后，其InitListen接口负责开启指定端口的监听。由于默认配置文件里GameNetDriver使用的是UIpNetDriver，所以这个开启监听的逻辑在UIpNetDriver::InitListen里:

bool UIpNetDriver::InitListen( FNetworkNotify* InNotify, FURL& LocalURL, bool bReuseAddressAndPort, FString& Error )
{
	if( !InitBase( false, InNotify, LocalURL, bReuseAddressAndPort, Error ) )
	{
		UE_LOG(LogNet, Warning, TEXT("Failed to init net driver ListenURL: %s: %s"), *LocalURL.ToString(), *Error);
		return false;
	}

	InitConnectionlessHandler();

	// Update result URL.
	//LocalURL.Host = LocalAddr->ToString(false);
	LocalURL.Port = LocalAddr->GetPort();
	UE_LOG(LogNet, Log, TEXT("%s IpNetDriver listening on port %i"), *GetDescription(), LocalURL.Port );

	return true;
}

这个InitListen函数的重点是InitBase，这个函数里面负责真正的创建Socket并在指定的端口进行监听:

bool UIpNetDriver::InitBase( bool bInitAsClient, FNetworkNotify* InNotify, const FURL& URL, bool bReuseAddressAndPort, FString& Error )
{
	using namespace UE::Net::Private;

	if (!Super::InitBase(bInitAsClient, InNotify, URL, bReuseAddressAndPort, Error))
	{
		return false;
	}

	ISocketSubsystem* SocketSubsystem = GetSocketSubsystem();
	if (SocketSubsystem == nullptr)
	{
		UE_LOG(LogNet, Warning, TEXT("Unable to find socket subsystem"));
		return false;
	}

	const int32 BindPort = bInitAsClient ? GetClientPort() : URL.Port;
	// Increase socket queue size, because we are polling rather than threading
	// and thus we rely on the OS socket to buffer a lot of data.
	const int32 DesiredRecvSize = bInitAsClient ? ClientDesiredSocketReceiveBufferBytes : ServerDesiredSocketReceiveBufferBytes;
	const int32 DesiredSendSize = bInitAsClient ? ClientDesiredSocketSendBufferBytes : ServerDesiredSocketSendBufferBytes;
	const EInitBindSocketsFlags InitBindFlags = bInitAsClient ? EInitBindSocketsFlags::Client : EInitBindSocketsFlags::Server;
	FCreateAndBindSocketFunc CreateAndBindSocketsFunc = [this, BindPort, bReuseAddressAndPort, DesiredRecvSize, DesiredSendSize]
									(TSharedRef<FInternetAddr> BindAddr, FString& Error) -> FUniqueSocket
		{
			return this->CreateAndBindSocket(BindAddr, BindPort, bReuseAddressAndPort, DesiredRecvSize, DesiredSendSize, Error);
		};

	bool bInitBindSocketsSuccess = Resolver->InitBindSockets(MoveTemp(CreateAndBindSocketsFunc), InitBindFlags, SocketSubsystem, Error);

	if (!bInitBindSocketsSuccess)
	{
		UE_LOG(LogNet, Error, TEXT("InitBindSockets failed: %s"), ToCStr(Error));

		return false;
	}

	
	// If the cvar is set and the socket subsystem supports it, create the receive thread.
	if (CVarNetIpNetDriverUseReceiveThread.GetValueOnAnyThread() != 0 && SocketSubsystem->IsSocketWaitSupported())
	{
		SocketReceiveThreadRunnable = MakeUnique<FReceiveThreadRunnable>(this);
		SocketReceiveThread.Reset(FRunnableThread::Create(SocketReceiveThreadRunnable.Get(), *FString::Printf(TEXT("IpNetDriver Receive Thread: %s"), *NetDriverName.ToString())));
	}

	SetSocketAndLocalAddress(Resolver->GetFirstSocket());

	bool bRecvMultiEnabled = CVarNetUseRecvMulti.GetValueOnAnyThread() != 0;
	bool bRecvThreadEnabled = CVarNetIpNetDriverUseReceiveThread.GetValueOnAnyThread() != 0;

	if (bRecvMultiEnabled && !bRecvThreadEnabled)
	{
		// 忽略多线程收发的处理
	}
	else if (bRecvMultiEnabled && bRecvThreadEnabled)
	{
		UE_LOG(LogNet, Warning, TEXT("NetDriver RecvMulti is not yet supported with the Receive Thread enabled."));
	}

	// Success.
	return true;
}

这里的CreateAndBindSocket是最终负责绑定的地方，由于指定的端口可能已经被使用了，所以这里内部会使用BindNextPort来不断递增从而获取下一个可以使用的端口号:

int32 ISocketSubsystem::BindNextPort(FSocket* Socket, FInternetAddr& Addr, int32 PortCount, int32 PortIncrement)
{
	// go until we reach the limit (or we succeed)
	for (int32 Index = 0; Index < PortCount; Index++)
	{
		// try to bind to the current port
		if (Socket->Bind(Addr) == true)
		{
			// if it succeeded, return the port
			if (Addr.GetPort() != 0)
			{
				return Addr.GetPort();
			}
			else
			{
				return Socket->GetPortNo();
			}
		}
		// if the address had no port, we are done
		if( Addr.GetPort() == 0 )
		{
			break;
		}

		// increment to the next port, and loop!
		Addr.SetPort(Addr.GetPort() + PortIncrement);
	}

	return 0;
}

FUniqueSocket UIpNetDriver::CreateAndBindSocket(TSharedRef<FInternetAddr> BindAddr, int32 Port, bool bReuseAddressAndPort, int32 DesiredRecvSize, int32 DesiredSendSize, FString& Error)
{
	ISocketSubsystem* SocketSubsystem = GetSocketSubsystem();
	if (SocketSubsystem == nullptr)
	{
		Error = TEXT("Unable to find socket subsystem");
		return nullptr;
	}

	// Create the socket that we will use to communicate with
	FUniqueSocket NewSocket = CreateSocketForProtocol(BindAddr->GetProtocolType());

	// 忽略一些参数设置代码

	int32 ActualRecvSize(0);
	int32 ActualSendSize(0);
	NewSocket->SetReceiveBufferSize(DesiredRecvSize, ActualRecvSize);
	NewSocket->SetSendBufferSize(DesiredSendSize, ActualSendSize);
	UE_LOG(LogInit, Log, TEXT("%s: Socket queue. Rx: %i (config %i) Tx: %i (config %i)"), SocketSubsystem->GetSocketAPIName(),
		ActualRecvSize, DesiredRecvSize, ActualSendSize, DesiredSendSize);

	// Bind socket to our port.
	BindAddr->SetPort(Port);

	int32 AttemptPort = BindAddr->GetPort();
	int32 BoundPort = SocketSubsystem->BindNextPort(NewSocket.Get(), *BindAddr, MaxPortCountToTry + 1, 1);
	if (BoundPort == 0)
	{
		Error = FString::Printf(TEXT("%s: binding to port %i failed (%i)"), SocketSubsystem->GetSocketAPIName(), AttemptPort,
			(int32)SocketSubsystem->GetLastErrorCode());

		if (bExitOnBindFailure)
		{
			UE_LOG(LogNet, Fatal, TEXT("Fatal error: %s"), *Error);
		}

		return nullptr;
	}

	if (NewSocket->SetNonBlocking() == false)
	{
		Error = FString::Printf(TEXT("%s: SetNonBlocking failed (%i)"), SocketSubsystem->GetSocketAPIName(),
			(int32)SocketSubsystem->GetLastErrorCode());
		return nullptr;
	}

	return NewSocket;
}

虽然UE的底层Socket系统同时支持了UDP和TCP的通信，但是这里的UIpNetDriver默认使用的是基于UDP的Socket通信:

FUniqueSocket UIpNetDriver::CreateSocketForProtocol(const FName& ProtocolType)
{
	// Create UDP socket and enable broadcasting.
	ISocketSubsystem* SocketSubsystem = GetSocketSubsystem();

	if (SocketSubsystem == NULL)
	{
		UE_LOG(LogNet, Warning, TEXT("UIpNetDriver::CreateSocket: Unable to find socket subsystem"));
		return NULL;
	}

	return SocketSubsystem->CreateUniqueSocket(NAME_DGram, TEXT("Unreal"), ProtocolType);
}

由于UDP是无连接的，所以不像TCP一样有accept连接建立回调，只能通过接收到的数据来区分发送端属于哪个逻辑连接。因此在UIpNetDriver::InitListen里会执行InitConnectionlessHandle来注册消息处理函数PacketHandler来应对这个无状态网络连接的入站消息:

void UNetDriver::InitConnectionlessHandler()
{
	check(!ConnectionlessHandler.IsValid());

#if !UE_BUILD_SHIPPING
	if (!FParse::Param(FCommandLine::Get(), TEXT("NoPacketHandler")))
#endif
	{
		ConnectionlessHandler = MakeUnique<PacketHandler>(&DDoS);

		if (ConnectionlessHandler.IsValid())
		{
			ConnectionlessHandler->NotifyAnalyticsProvider(AnalyticsProvider, AnalyticsAggregator);
			ConnectionlessHandler->Initialize(UE::Handler::Mode::Server, MAX_PACKET_SIZE, true, nullptr, nullptr, NetDriverDefinition);

			// Add handling for the stateless connect handshake, for connectionless packets, as the outermost layer
			TSharedPtr<HandlerComponent> NewComponent =
				ConnectionlessHandler->AddHandler(TEXT("Engine.EngineHandlerComponentFactory(StatelessConnectHandlerComponent)"), true);

			StatelessConnectComponent = StaticCastSharedPtr<StatelessConnectHandlerComponent>(NewComponent);

			if (StatelessConnectComponent.IsValid())
			{
				StatelessConnectComponent.Pin()->SetDriver(this);
			}

			ConnectionlessHandler->InitializeComponents();
		}
	}
}

在这个函数里会注册一个StatelessConnectHandlerComponent来处理逻辑层网络连接的握手。

连接建立

当客户端准备连接到指定服务器的时候，需要以某种途径获取服务器的URL，然后通过UEngine::Browse来触发客户端的NetDriver初始化:

EBrowseReturnVal::Type UEngine::Browse( FWorldContext& WorldContext, FURL URL, FString& Error )
{
	Error = TEXT("");
	WorldContext.TravelURL = TEXT("");

	UE_LOGSTATUS(Log, TEXT("Started Browse: \"%s\""), *URL.ToString());
	// 省略很多代码
	if( URL.IsLocalInternal() )
	{
		// Local map file.
		return LoadMap( WorldContext, URL, NULL, Error ) ? EBrowseReturnVal::Success : EBrowseReturnVal::Failure;
	}
	else if( URL.IsInternal() && GIsClient )
	{
		// Network URL.
		if( WorldContext.PendingNetGame )
		{
			CancelPending(WorldContext);
		}

		// Clean up the netdriver/socket so that the pending level succeeds
		if (WorldContext.World() && ShouldShutdownWorldNetDriver())
		{
			ShutdownWorldNetDriver(WorldContext.World());
		}

		WorldContext.PendingNetGame = NewObject<UPendingNetGame>();
		WorldContext.PendingNetGame->Initialize(URL); //-V595
		WorldContext.PendingNetGame->InitNetDriver(); //-V595
		// 省略很多代码
		return EBrowseReturnVal::Pending;
	}
}

这里会创建一个UPendingNetGame对象来处理客户端到服务端连接正式建立之前的一些握手逻辑，首先就是创建一个GameNetDriver，也就是之前介绍过的UIpNetDriver:

void UPendingNetGame::InitNetDriver()
{
	LLM_SCOPE(ELLMTag::Networking);

	if (!GDisallowNetworkTravel)
	{
		NETWORK_PROFILER(GNetworkProfiler.TrackSessionChange(true, URL));

		// Try to create network driver.
		if (GEngine->CreateNamedNetDriver(this, NAME_PendingNetDriver, NAME_GameNetDriver))
		{
			NetDriver = GEngine->FindNamedNetDriver(this, NAME_PendingNetDriver);
		}

		if (NetDriver == nullptr)
		{
			UE_LOG(LogNet, Warning, TEXT("Error initializing the pending net driver.  Check the configuration of NetDriverDefinitions and make sure module/plugin dependencies are correct."));
			ConnectionError = NSLOCTEXT("Engine", "NetworkDriverInit", "Error creating network driver.").ToString();
			return;
		}

		if( NetDriver->InitConnect( this, URL, ConnectionError ) )
		{
			FNetDelegates::OnPendingNetGameConnectionCreated.Broadcast(this);

			ULocalPlayer* LocalPlayer = GEngine->GetFirstGamePlayer(this);
			if (LocalPlayer)
			{
				LocalPlayer->PreBeginHandshake(ULocalPlayer::FOnPreBeginHandshakeCompleteDelegate::CreateWeakLambda(this,
					[this]()
					{
						BeginHandshake();
					}));
			}
			else
			{
				BeginHandshake();
			}
		}
		else
		{
			// 忽略一些错误处理代码
		}
	}
}

在NetDriver被创建之后，接下来使用InitConnect来向服务端URL发起一个连接建立请求,这里的InitBase我们在前面已经介绍过了，会注册一个UDP端口的监听，同时注册StatelessConnectHandlerComponent为消息处理函数:

bool UIpNetDriver::InitConnect( FNetworkNotify* InNotify, const FURL& ConnectURL, FString& Error )
{
	using namespace UE::Net::Private;

	ISocketSubsystem* SocketSubsystem = GetSocketSubsystem();
	if (SocketSubsystem == nullptr)
	{
		UE_LOG(LogNet, Warning, TEXT("Unable to find socket subsystem"));
		return false;
	}

	if( !InitBase( true, InNotify, ConnectURL, false, Error ) )
	{
		UE_LOG(LogNet, Warning, TEXT("Failed to init net driver ConnectURL: %s: %s"), *ConnectURL.ToString(), *Error);
		return false;
	}

	// Create new connection.
	ServerConnection = NewObject<UNetConnection>(GetTransientPackage(), NetConnectionClass);

	ServerConnection->InitLocalConnection(this, SocketPrivate.Get(), ConnectURL, USOCK_Pending);

	Resolver->InitConnect(ServerConnection, SocketSubsystem, GetSocket(), ConnectURL);

	UIpConnection* IpServerConnection = Cast<UIpConnection>(ServerConnection);

	if (FNetConnectionAddressResolution* ConnResolver = FNetDriverAddressResolution::GetConnectionResolver(IpServerConnection))
	{
		if (ConnResolver->IsAddressResolutionEnabled() && !ConnResolver->IsAddressResolutionComplete())
		{
			SocketState = ESocketState::Resolving;
		}
	}
	
	UE_LOG(LogNet, Log, TEXT("Game client on port %i, rate %i"), ConnectURL.Port, ServerConnection->CurrentNetSpeed );
	CreateInitialClientChannels();

	return true;
}

在这里终于看到逻辑层连接对象ServerConnection，这个对象会在InitConnect的时候被创建，创建之后的Resolver->InitConnect负责发起执行服务端URL的DNS查询，同时这里的SocketState的状态会被标记为ESocketState::Resolving。最后面的CreateInitialClientChannels会创建一些初始的Channel，每个Channel相当于一个信道，往同一个信道里投递的可靠消息能保证被对端有序接收：

void UNetDriver::CreateInitialClientChannels()
{
	if (ServerConnection != nullptr)
	{
		for (const FChannelDefinition& ChannelDef : ChannelDefinitions)
		{
			if (ChannelDef.bInitialClient && (ChannelDef.ChannelClass != nullptr))
			{
				ServerConnection->CreateChannelByName(ChannelDef.ChannelName, EChannelCreateFlags::OpenedLocally, ChannelDef.StaticChannelIndex);
			}
		}
	}
}

在默认配置里，会配置下面四个Channel:

[/Script/Engine.NetDriver]
+ChannelDefinitions=(ChannelName=Control, ClassName=/Script/Engine.ControlChannel, StaticChannelIndex=0, bTickOnCreate=true, bServerOpen=false, bClientOpen=true, bInitialServer=false, bInitialClient=true)
+ChannelDefinitions=(ChannelName=Voice, ClassName=/Script/Engine.VoiceChannel, StaticChannelIndex=1, bTickOnCreate=true, bServerOpen=true, bClientOpen=true, bInitialServer=true, bInitialClient=true)
+ChannelDefinitions=(ChannelName=DataStream, ClassName=/Script/Engine.DataStreamChannel, StaticChannelIndex=2, bTickOnCreate=true, bServerOpen=true, bClientOpen=true, bInitialServer=true, bInitialClient=true)
+ChannelDefinitions=(ChannelName=Actor, ClassName=/Script/Engine.ActorChannel, StaticChannelIndex=-1, bTickOnCreate=false, bServerOpen=true, bClientOpen=false, bInitialServer=false, bInitialClient=false)

当UIpNetDriver::InitConnect结束之后，控制流回到了UPendingNetGame::InitNetDriver，这里开始正式的向服务器发起handshake:

void UPendingNetGame::BeginHandshake()
{
	// Kick off the connection handshake
	UNetConnection* ServerConn = NetDriver->ServerConnection;
	if (ServerConn->Handler.IsValid())
	{
		ServerConn->Handler->BeginHandshaking(
			FPacketHandlerHandshakeComplete::CreateUObject(this, &UPendingNetGame::SendInitialJoin));
	}
	else
	{
		SendInitialJoin();
	}
}

这里的serverConn->Handler对象就是之前的PacketHandler，在PacketHandler::BeginHandShaking里会遍历所有的注册过来的HandlerComponent来执行握手操作:

void PacketHandler::BeginHandshaking(FPacketHandlerHandshakeComplete InHandshakeDel/*=FPacketHandlerHandshakeComplete()*/)
{
	check(!bBeganHandshaking);

	bBeganHandshaking = true;

	HandshakeCompleteDel = InHandshakeDel;

	for (int32 i=HandlerComponents.Num() - 1; i>=0; --i)
	{
		HandlerComponent& CurComponent = *HandlerComponents[i];

		if (CurComponent.RequiresHandshake() && !CurComponent.IsInitialized())
		{
			CurComponent.NotifyHandshakeBegin();
			break;
		}
	}
}

这里我们只需要关心之前创建的StatelessConnectHandlerComponent相关握手逻辑:

void StatelessConnectHandlerComponent::NotifyHandshakeBegin()
{
	using namespace UE::Net;

	SendInitialPacket(static_cast<EHandshakeVersion>(CurrentHandshakeVersion));
}
void StatelessConnectHandlerComponent::SendInitialPacket(EHandshakeVersion HandshakeVersion)
{
	using namespace UE::Net;

	if (Handler->Mode == UE::Handler::Mode::Client)
	{
		UNetConnection* ServerConn = (Driver != nullptr ? ToRawPtr(Driver->ServerConnection) : nullptr);

		if (ServerConn != nullptr)
		{
			const int32 AdjustedSize = GetAdjustedSizeBits(HANDSHAKE_PACKET_SIZE_BITS, HandshakeVersion);
			FBitWriter InitialPacket(AdjustedSize + (BaseRandomDataLengthBytes * 8) + 1 /* Termination bit */);

			BeginHandshakePacket(InitialPacket, EHandshakePacketType::InitialPacket, HandshakeVersion, SentHandshakePacketCount, CachedClientID,
									(bRestartedHandshake ? EHandshakePacketModifier::RestartHandshake : EHandshakePacketModifier::None));

			uint8 SecretIdPad = 0;
			uint8 PacketSizeFiller[28];

			InitialPacket.WriteBit(SecretIdPad);

			FMemory::Memzero(PacketSizeFiller, UE_ARRAY_COUNT(PacketSizeFiller));
			InitialPacket.Serialize(PacketSizeFiller, UE_ARRAY_COUNT(PacketSizeFiller));

			SendToServer(HandshakeVersion, EHandshakePacketType::InitialPacket, InitialPacket);
		}
		else
		{
			UE_LOG(LogHandshake, Error, TEXT("Tried to send handshake connect packet without a server connection."));
		}
	}
}

这里的InitialPacket的具体格式在源代码文件里的注释里写的非常清楚:

 * Handshake Process/Protocol:
 * --------------------------
 *
 * The protocol for the handshake involves the client sending an initial packet to the server,
 * and the server responding with a unique 'Cookie' value, which the client has to respond with.
 *
 * Client - Initial Connect:
 *
 * [?:MagicHeader][2:SessionID][3:ClientID][HandshakeBit][RestartHandshakeBit]
 * [8:MinVersion][8:CurVersion][8:HandshakePacketType][8:SentPacketCount][32:NetworkVersion]
 * [16:NetworkFeatures][SecretIdBit][28:PacketSizeFiller][AlignPad][?:RandomData]

填充好InitialPacket之后，调用SendToServer往服务器端发送这个握手消息，这里会有一个比较特殊的标记SetRawSend，作用是发送消息的时候直接走底层的发送接口，不要被上层的Handler托管:

void StatelessConnectHandlerComponent::SendToServer(EHandshakeVersion HandshakeVersion, EHandshakePacketType PacketType, FBitWriter& Packet)
{
	if (UNetConnection* ServerConn = (Driver != nullptr ? Driver->ServerConnection : nullptr))
	{
		CapHandshakePacket(Packet, HandshakeVersion);


		// Disable PacketHandler parsing, and send the raw packet
		Handler->SetRawSend(true);

		{
			if (Driver->IsNetResourceValid())
			{
				FOutPacketTraits Traits;

				Driver->ServerConnection->LowLevelSend(Packet.GetData(), Packet.GetNumBits(), Traits);
			}
		}

		Handler->SetRawSend(false);

		LastClientSendTimestamp = FPlatformTime::Seconds();
	}
}

/**
 * Sets whether or not outgoing packets should bypass this handler - used when raw packet sends are necessary
 * (such as for the stateless handshake)
 *
 * @param bInEnabled	Whether or not raw sends are enabled
 */
FORCEINLINE void SetRawSend(bool bInEnabled)
{
	bRawSend = bInEnabled;
}

由于当前的UIpNetDriver使用的是UDP通信，所以有可能出现消息丢失的问题。为了处理可能的握手消息丢失的情况，StatelessConnectHandlerComponent的Tick函数里会检查之前发送的握手包是不是超时了，如果超时了则再次执行SendInitialPacket:

void StatelessConnectHandlerComponent::Tick(float DeltaTime)
{
	using namespace UE::Net;

	if (Handler->Mode == UE::Handler::Mode::Client)
	{
		if (State != UE::Handler::Component::State::Initialized && LastClientSendTimestamp != 0.0)
		{
			double LastSendTimeDiff = FPlatformTime::Seconds() - LastClientSendTimestamp;

			if (LastSendTimeDiff > UE::Net::HandshakeResendInterval)
			{
				const bool bRestartChallenge = Driver != nullptr && ((Driver->GetElapsedTime() - LastChallengeTimestamp) > MIN_COOKIE_LIFETIME);

				if (bRestartChallenge)
				{
					SetState(UE::Handler::Component::State::UnInitialized);
				}

				if (State == UE::Handler::Component::State::UnInitialized)
				{
					UE_LOG(LogHandshake, Verbose, TEXT("Initial handshake packet timeout - resending."));

					EHandshakeVersion ResendVersion = static_cast<EHandshakeVersion>(CurrentHandshakeVersion);

					// 忽略一些无关代码

					SendInitialPacket(ResendVersion);
				}
				else if (State == UE::Handler::Component::State::InitializedOnLocal && LastTimestamp != 0.0)
				{
					UE_LOG(LogHandshake, Verbose, TEXT("Challenge response packet timeout - resending."));

					SendChallengeResponse(LastRemoteHandshakeVersion, LastSecretId, LastTimestamp, LastCookie);
				}
			}
		}
	}
	else
	{
		// 省略服务器相关逻辑
	}
}

UE接收网络数据的位置在UIpNetDriver::TickDispatch里，这个函数每帧都会被调用，这个函数里处理的逻辑太多了，这里就先考虑服务端处理新的客户端连接相关内容:

void UIpNetDriver::TickDispatch(float DeltaTime)
{
	LLM_SCOPE_BYTAG(NetDriver);

	Super::TickDispatch( DeltaTime );

	// Process all incoming packets
	for (FPacketIterator It(this); It; ++It)
	{
		FReceivedPacketView ReceivedPacket;
		FInPacketTraits& ReceivedTraits = ReceivedPacket.Traits;
		bool bOk = It.GetCurrentPacket(ReceivedPacket);
		const TSharedRef<const FInternetAddr> FromAddr = ReceivedPacket.Address.ToSharedRef();
		UNetConnection* Connection = nullptr;
		UIpConnection* const MyServerConnection = GetServerConnection();
		bool bIgnorePacket = false;

		// If we didn't find a client connection, maybe create a new one.
		if (Connection == nullptr)
		{
			
			// Determine if allowing for client/server connections
			const bool bAcceptingConnection = Notify != nullptr && Notify->NotifyAcceptingConnection() == EAcceptConnection::Accept;

			if (bAcceptingConnection)
			{
				if (!DDoS.CheckLogRestrictions() && !bExceededIPAggregationLimit)
				{
					TrackAndLogNewIP(FromAddr.Get());
				}

				FPacketBufferView WorkingBuffer = It.GetWorkingBuffer();

				Connection = ProcessConnectionlessPacket(ReceivedPacket, WorkingBuffer);
				bIgnorePacket = ReceivedPacket.DataView.NumBytes() == 0;
			}
		}
	}
}

这个函数会使用FPacketIterator来遍历当前Socket上接收到的网络数据，业务层数据包的最小粒度就是Packet。如果这个Packet的客户端地址并没有绑定好的UNetConnection，则可以认为这个Packet一定是来自于新客户端的相关数据包，这里会使用ProcessConnectionlessPacket来处理这些包:

UNetConnection* UIpNetDriver::ProcessConnectionlessPacket(FReceivedPacketView& PacketRef, const FPacketBufferView& WorkingBuffer)
{
	UNetConnection* ReturnVal = nullptr;
	TSharedPtr<StatelessConnectHandlerComponent> StatelessConnect;
	const TSharedPtr<const FInternetAddr>& Address = PacketRef.Address;
	FString IncomingAddress = Address->ToString(true);
	bool bPassedChallenge = false;
	bool bRestartedHandshake = false;
	bool bIgnorePacket = true;

	if (Notify != nullptr && ConnectionlessHandler.IsValid() && StatelessConnectComponent.IsValid())
	{
		StatelessConnect = StatelessConnectComponent.Pin();

		EIncomingResult Result = ConnectionlessHandler->IncomingConnectionless(PacketRef);
		// 省略后续所有代码
	}
	// 省略后续所有代码
}

每个包都会通过ConnectionlessHandler->IncomingConnectionless的处理，这个函数会最终调用到StatelessConnectHandlerComponent::IncomingConnectionless，这个函数里会重点处理handshake包逻辑:

void StatelessConnectHandlerComponent::IncomingConnectionless(FIncomingPacketRef PacketRef)
{
	using namespace UE::Net;

	FBitReader& Packet = PacketRef.Packet;
	const TSharedPtr<const FInternetAddr> Address = PacketRef.Address;

	if (MagicHeader.Num() > 0)
	{
		uint32 ReadMagic = 0;

		Packet.SerializeBits(&ReadMagic, MagicHeader.Num());

		if (GVerifyMagicHeader && ReadMagic != MagicHeaderUint)
		{
#if !UE_BUILD_SHIPPING
			UE_CLOG(TrackValidationLogs(), LogNet, Log, TEXT("Rejecting packet with invalid magic header '%08X' vs '%08X' (%i bits)"),
					ReadMagic, MagicHeaderUint, MagicHeader.Num());
#endif

			Packet.SetError();

			return;
		}
	}


	bool bHasValidSessionID = true;
	uint8 SessionID = 0;
	uint8 ClientID = 0;

	if (CurrentHandshakeVersion >= static_cast<uint8>(EHandshakeVersion::SessionClientId))
	{
		Packet.SerializeBits(&SessionID, SessionIDSizeBits);
		Packet.SerializeBits(&ClientID, ClientIDSizeBits);

		bHasValidSessionID = GVerifyNetSessionID == 0 || (SessionID == CachedGlobalNetTravelCount && !Packet.IsError());

		// No ClientID validation until connected
	}

	const bool bHandshakePacket = !!Packet.ReadBit() && !Packet.IsError();

	LastChallengeSuccessAddress = nullptr;

	// 忽略一些容错代码

	FParsedHandshakeData HandshakeData;
	const bool bValidHandshakePacket = ParseHandshakePacket(Packet, HandshakeData);

	//忽略bValidHandshakePacket为false的处理的代码
	
	const bool bIsServer = Handler->Mode == UE::Handler::Mode::Server;
	if (UNLIKELY(!bIsServer))
	{
		// Only server can negotiate handshake requests here
		return;
	}

	EHandshakeVersion TargetVersion = EHandshakeVersion::Latest;
	const bool bValidVersion = CheckVersion(HandshakeData, TargetVersion);
	const bool bInitialConnect = HandshakeData.HandshakePacketType == EHandshakePacketType::InitialPacket && HandshakeData.Timestamp == 0.0;
	const double ElapsedTime = Driver ? Driver->GetElapsedTime() : 0.0;

	const bool bIsValidRequest = bValidVersion && (bHasValidSessionID || bInitialConnect);

	// Handle invalid requests
	if (!bIsValidRequest)
	{
		// 忽略错误的包数据的处理
		return;
	}

	if (bInitialConnect)
	{
		SendConnectChallenge(FCommonSendToClientParams(Address, TargetVersion, ClientID), HandshakeData.RemoteSentHandshakePacketCount);
	}
	// 省略后续所有代码
}

上面的函数负责解析和验证一个有效的HandshakePacket，如果这个HandshakePacket是有效的，则会使用SendConnectChallenge来通知客户端:

void StatelessConnectHandlerComponent::SendConnectChallenge(FCommonSendToClientParams CommonParams, uint8 ClientSentHandshakePacketCount)
{
	using namespace UE::Net;

	if (Driver != nullptr)
	{
		const int32 AdjustedSize = GetAdjustedSizeBits(HANDSHAKE_PACKET_SIZE_BITS, CommonParams.HandshakeVersion);
		FBitWriter ChallengePacket(AdjustedSize + (BaseRandomDataLengthBytes * 8) + 1 /* Termination bit */);

		BeginHandshakePacket(ChallengePacket, EHandshakePacketType::Challenge, CommonParams.HandshakeVersion, ClientSentHandshakePacketCount,
								CommonParams.ClientID);

		double Timestamp = Driver->GetElapsedTime();
		uint8 Cookie[COOKIE_BYTE_SIZE];

		GenerateCookie(CommonParams.ClientAddress, ActiveSecret, Timestamp, Cookie);

		ChallengePacket.WriteBit(ActiveSecret);

		ChallengePacket << Timestamp;

		ChallengePacket.Serialize(Cookie, UE_ARRAY_COUNT(Cookie));

		SendToClient(CommonParams, EHandshakePacketType::Challenge, ChallengePacket);
	}
}

这个SendConnectChallenge的作用是生成一个Challenge包，包里会塞入当前时间戳和一个随机生成的与这个ClientAddress所绑定的Cookie，作为临时的Session标识符来使用。这个Packet填充完成之后，使用SendToClient发送回对应的客户端地址,这个SendToClient的具体实现与之前介绍的SendToServer的实现基本一样，都是使用底层的LowLevelSend来跳过PacketHandler直接调用系统接口来执行数据发送。

接下来要考虑的是客户端如何接收这个Challenge包，虽然数据接收的地方依然是在UIpNetDriver::TickDispatch,但是由于此时客户端早就建立好了与服务端之间的网络连接对象，因此走的代码分支是不一样的:

void UIpNetDriver::TickDispatch(float DeltaTime)
{
	LLM_SCOPE_BYTAG(NetDriver);

	Super::TickDispatch( DeltaTime );
	// Process all incoming packets
	for (FPacketIterator It(this); It; ++It)
	{
		FReceivedPacketView ReceivedPacket;
		FInPacketTraits& ReceivedTraits = ReceivedPacket.Traits;
		bool bOk = It.GetCurrentPacket(ReceivedPacket);
		const TSharedRef<const FInternetAddr> FromAddr = ReceivedPacket.Address.ToSharedRef();
		UNetConnection* Connection = nullptr;
		UIpConnection* const MyServerConnection = GetServerConnection();
		// Figure out which socket the received data came from.
		if (MyServerConnection)
		{
			if (MyServerConnection->RemoteAddr->CompareEndpoints(*FromAddr))
			{
				Connection = MyServerConnection;
			}
		}
		// Send the packet to the connection for processing.
		if (Connection != nullptr && !bIgnorePacket)
		{
			if (DDoS.IsDDoSDetectionEnabled())
			{
				DDoS.IncNetConnPacketCounter();
				DDoS.CondCheckNetConnLimits();
			}

			if (bRetrieveTimestamps)
			{
				It.GetCurrentPacketTimestamp(Connection);
			}

			Connection->ReceivedRawPacket((uint8*)ReceivedPacket.DataView.GetData(), ReceivedPacket.DataView.NumBytes());
		}
	}
}

客户端处理消息的时候会将Packet的来源地址与当前记录的服务端连接MyServerConnection的地址相比较，如果相等则会直接使用这个Connection的ReceivedRawPacket来处理当前的ReceivedPacket:

void UNetConnection::ReceivedRawPacket( void* InData, int32 Count )
{
	using namespace UE::Net;

	uint8* Data = (uint8*)InData;

	++InTotalHandlerPackets;

	if (Handler.IsValid())
	{
		FReceivedPacketView PacketView;

		PacketView.DataView = {Data, Count, ECountUnits::Bytes};

		EIncomingResult IncomingResult = Handler->Incoming(PacketView);
		// 省略后续代码
	}
	// 省略后续代码
}

这个ReceivedRawPacket又会调用到Handler->Incoming,这个函数是所有包处理的通用接口，这里重点关注Challenge包的逻辑处理，也就是下面的bIsChallengePacket=true的部分:

void StatelessConnectHandlerComponent::Incoming(FBitReader& Packet)
{
	using namespace UE::Net;

	if (MagicHeader.Num() > 0)
	{
		// Don't bother with the expense of verifying the magic header here.
		uint32 ReadMagic = 0;
		Packet.SerializeBits(&ReadMagic, MagicHeader.Num());
	}

	bool bHasValidSessionID = true;
	bool bHasValidClientID = true;
	uint8 SessionID = 0;
	uint8 ClientID = 0;

	if (LastRemoteHandshakeVersion >= EHandshakeVersion::SessionClientId)
	{
		Packet.SerializeBits(&SessionID, SessionIDSizeBits);
		Packet.SerializeBits(&ClientID, ClientIDSizeBits);

		bHasValidSessionID = GVerifyNetSessionID == 0 || (SessionID == CachedGlobalNetTravelCount && !Packet.IsError());
		bHasValidClientID = GVerifyNetClientID == 0 || (ClientID == CachedClientID && !Packet.IsError());
	}

	bool bHandshakePacket = !!Packet.ReadBit() && !Packet.IsError();

	if (bHandshakePacket)
	{
		FParsedHandshakeData HandshakeData;

		bHandshakePacket = ParseHandshakePacket(Packet, HandshakeData);

		if (bHandshakePacket)
		{
			const bool bIsChallengePacket = HandshakeData.HandshakePacketType == EHandshakePacketType::Challenge && HandshakeData.Timestamp > 0.0;
			const bool bIsInitialChallengePacket = bIsChallengePacket && State != UE::Handler::Component::State::Initialized;
			const bool bIsUpgradePacket = HandshakeData.HandshakePacketType == EHandshakePacketType::VersionUpgrade;

			if (Handler->Mode == UE::Handler::Mode::Client && bHasValidClientID && (bHasValidSessionID || bIsInitialChallengePacket || bIsUpgradePacket))
			{
				if (State == UE::Handler::Component::State::UnInitialized || State == UE::Handler::Component::State::InitializedOnLocal)
				{
					if (HandshakeData.bRestartHandshake)
					{
#if !UE_BUILD_SHIPPING
						UE_LOG(LogHandshake, Log, TEXT("Ignoring restart handshake request, while already restarted."));
#endif
					}
					// Receiving challenge
					else if (bIsChallengePacket)
					{
#if !UE_BUILD_SHIPPING
						UE_LOG(LogHandshake, Log, TEXT("Cached server SessionID: %u"), SessionID);
#endif

						CachedGlobalNetTravelCount = SessionID;

						LastChallengeTimestamp = (Driver != nullptr ? Driver->GetElapsedTime() : 0.0);

						SendChallengeResponse(HandshakeData.RemoteCurVersion, HandshakeData.SecretId, HandshakeData.Timestamp, HandshakeData.Cookie);

						// Utilize this state as an intermediary, indicating that the challenge response has been sent
						SetState(UE::Handler::Component::State::InitializedOnLocal);
						// 省略后续所有代码
					}
				}
			}
		}
	}
}

bIsChallengePacket的处理结果就是客户端会通过SendChallengeResponse发送一个ChallengeResponse包到服务器,这里会将服务端Challenge包里的SecretID,Cookie,Timestamp等字段再次打包进去:

void StatelessConnectHandlerComponent::SendChallengeResponse(EHandshakeVersion HandshakeVersion, uint8 InSecretId, double InTimestamp,
																uint8 InCookie[COOKIE_BYTE_SIZE])
{
	using namespace UE::Net;

	UNetConnection* ServerConn = (Driver != nullptr ? ToRawPtr(Driver->ServerConnection) : nullptr);

	if (ServerConn != nullptr)
	{
		const int32 AdjustedSize = GetAdjustedSizeBits((bRestartedHandshake ? RESTART_RESPONSE_SIZE_BITS : HANDSHAKE_PACKET_SIZE_BITS),
														HandshakeVersion);
		FBitWriter ResponsePacket(AdjustedSize + (BaseRandomDataLengthBytes * 8) + 1 /* Termination bit */);
		EHandshakePacketType HandshakePacketType = bRestartedHandshake ? EHandshakePacketType::RestartResponse : EHandshakePacketType::Response;

		BeginHandshakePacket(ResponsePacket, HandshakePacketType, HandshakeVersion, SentHandshakePacketCount, CachedClientID,
								(bRestartedHandshake ? EHandshakePacketModifier::RestartHandshake : EHandshakePacketModifier::None));

		ResponsePacket.WriteBit(InSecretId);

		ResponsePacket << InTimestamp;
		ResponsePacket.Serialize(InCookie, COOKIE_BYTE_SIZE);

		if (bRestartedHandshake)
		{
			ResponsePacket.Serialize(AuthorisedCookie, COOKIE_BYTE_SIZE);
		}

#if !UE_BUILD_SHIPPING
		UE_LOG(LogHandshake, Log, TEXT("SendChallengeResponse. Timestamp: %f, Cookie: %s"), InTimestamp,
				*FString::FromBlob(InCookie, COOKIE_BYTE_SIZE));
#endif

		SendToServer(HandshakeVersion, HandshakePacketType, ResponsePacket);


		int16* CurSequence = (int16*)InCookie;

		LastSecretId = InSecretId;
		LastTimestamp = InTimestamp;
		LastServerSequence = *CurSequence & (MAX_PACKETID - 1);
		LastClientSequence = *(CurSequence + 1) & (MAX_PACKETID - 1);
		LastRemoteHandshakeVersion = HandshakeVersion;

		FMemory::Memcpy(LastCookie, InCookie, UE_ARRAY_COUNT(LastCookie));
	}
	else
	{
		UE_LOG(LogHandshake, Error, TEXT("Tried to send handshake response packet without a server connection."));
	}
}

这里在使用SendToServer发送完数据之后，会将Cookie的头两个int16数据当作服务端客户端通信的包序列号，这样强制转换的作用是为了随机初始化第一个包的序列号。

数据走到服务端之后，仍然会调用到StatelessConnectHandlerComponent::IncomingConnectionless来处理，同样的这里我们只关心ChallengeResponse的部分，也就是bInitialConnect=false的分支:

void StatelessConnectHandlerComponent::IncomingConnectionless(FIncomingPacketRef PacketRef)
{
	using namespace UE::Net;

	FBitReader& Packet = PacketRef.Packet;
	const TSharedPtr<const FInternetAddr> Address = PacketRef.Address;
	// 省略很多代码
	if (bInitialConnect)
	{
		SendConnectChallenge(FCommonSendToClientParams(Address, TargetVersion, ClientID), HandshakeData.RemoteSentHandshakePacketCount);
	}
	else
	{
		// Challenge response
		// NOTE: Allow CookieDelta to be 0.0, as it is possible for a server to send a challenge and receive a response,
		//			during the same tick
		bool bChallengeSuccess = false;
		const double CookieDelta = ElapsedTime - HandshakeData.Timestamp;
		const double SecretDelta = HandshakeData.Timestamp - LastSecretUpdateTimestamp;
		const bool bValidCookieLifetime = CookieDelta >= 0.0 && (MAX_COOKIE_LIFETIME - CookieDelta) > 0.0;
		const bool bValidSecretIdTimestamp = (HandshakeData.SecretId == ActiveSecret) ? (SecretDelta >= 0.0) : (SecretDelta <= 0.0);

		if (bValidCookieLifetime && bValidSecretIdTimestamp)
		{
			// Regenerate the cookie from the packet info, and see if the received cookie matches the regenerated one
			uint8 RegenCookie[COOKIE_BYTE_SIZE];

			GenerateCookie(Address, HandshakeData.SecretId, HandshakeData.Timestamp, RegenCookie);

			bChallengeSuccess = FMemory::Memcmp(HandshakeData.Cookie, RegenCookie, COOKIE_BYTE_SIZE) == 0;

			if (bChallengeSuccess)
			{
				if (HandshakeData.bRestartHandshake)
				{
					FMemory::Memcpy(AuthorisedCookie, HandshakeData.OrigCookie, UE_ARRAY_COUNT(AuthorisedCookie));
				}
				else
				{
					int16* CurSequence = (int16*)HandshakeData.Cookie;

					LastServerSequence = *CurSequence & (MAX_PACKETID - 1);
					LastClientSequence = *(CurSequence + 1) & (MAX_PACKETID - 1);

					FMemory::Memcpy(AuthorisedCookie, HandshakeData.Cookie, UE_ARRAY_COUNT(AuthorisedCookie));
				}

				bRestartedHandshake = HandshakeData.bRestartHandshake;
				LastChallengeSuccessAddress = Address->Clone();
				LastRemoteHandshakeVersion = TargetVersion;
				CachedClientID = ClientID;

				if (TargetVersion < MinClientHandshakeVersion && static_cast<uint8>(TargetVersion) >= MinSupportedHandshakeVersion)
				{
					MinClientHandshakeVersion = TargetVersion;
				}

				// Now ack the challenge response - the cookie is stored in AuthorisedCookie, to enable retries
				SendChallengeAck(FCommonSendToClientParams(Address, TargetVersion, ClientID), HandshakeData.RemoteSentHandshakePacketCount, AuthorisedCookie);
			}
		}
	}
}

在这里会利用时间戳和客户端地址重新生成一次Cookie，如果客户端发送过来的包里携带的Cookie与此时生成的RegenCookie一致，则代表ChallengeResponse被成功接收，此时也会从Cookie的前两个int16转换为客户端与服务端两者通信的包序列号，这样就实现了上行包序列号与下行包序列号的第一次同步。最后再发送一个ChallengeAck包通知到客户端连接建立完成。在StatelessConnectHandlerComponent::IncomingConnectionless处理完成了Challenge的验证之后，其外层调用UIpNetDriver::ProcessConnectionlessPacket会再次检查是否Challenge接收成功，如果成功了则会创建一个对应的客户端连接UIpConnection:

UNetConnection* UIpNetDriver::ProcessConnectionlessPacket(FReceivedPacketView& PacketRef, const FPacketBufferView& WorkingBuffer)
{
	UNetConnection* ReturnVal = nullptr;
	TSharedPtr<StatelessConnectHandlerComponent> StatelessConnect;
	const TSharedPtr<const FInternetAddr>& Address = PacketRef.Address;
	FString IncomingAddress = Address->ToString(true);
	bool bPassedChallenge = false;
	bool bRestartedHandshake = false;
	bool bIgnorePacket = true;

	if (Notify != nullptr && ConnectionlessHandler.IsValid() && StatelessConnectComponent.IsValid())
	{
		StatelessConnect = StatelessConnectComponent.Pin();

		EIncomingResult Result = ConnectionlessHandler->IncomingConnectionless(PacketRef);

		if (Result == EIncomingResult::Success)
		{
			bPassedChallenge = StatelessConnect->HasPassedChallenge(Address, bRestartedHandshake);

			// 省略一些代码
		}
	}
	if (bPassedChallenge)
	{
		if (!bRestartedHandshake)
		{
			SCOPE_CYCLE_COUNTER(Stat_IpNetDriverAddNewConnection);

			UE_LOG(LogNet, Log, TEXT("Server accepting post-challenge connection from: %s"), *IncomingAddress);

			ReturnVal = NewObject<UIpConnection>(GetTransientPackage(), NetConnectionClass);
			check(ReturnVal != nullptr);

			ReturnVal->InitRemoteConnection(this, SocketPrivate.Get(), World ? World->URL : FURL(), *Address, USOCK_Open);

			// Set the initial packet sequence from the handshake data
			if (StatelessConnect.IsValid())
			{
				int32 ServerSequence = 0;
				int32 ClientSequence = 0;

				StatelessConnect->GetChallengeSequence(ServerSequence, ClientSequence);

				ReturnVal->InitSequence(ClientSequence, ServerSequence);
			}

			if (ReturnVal->Handler.IsValid())
			{
				ReturnVal->Handler->BeginHandshaking();
			}

			Notify->NotifyAcceptedConnection(ReturnVal);

			AddClientConnection(ReturnVal);
			RemoveFromNewIPTracking(*Address.Get());
		}

		if (StatelessConnect.IsValid())
		{
			StatelessConnect->ResetChallengeData();
		}
	}
	// 省略后续代码
}

这个ClientConnection被创建之后，会利用StatelessConnect里的数据做一些字段初始化，例如ServerSequence,ClientSequence。初始化完成之后还会通过Notify来通知到当前的World新的客户端连接:

void UWorld::NotifyAcceptedConnection( UNetConnection* Connection )
{
	check(NetDriver!=NULL);
	check(NetDriver->ServerConnection==NULL);
	UE_LOG(LogNet, Log, TEXT("NotifyAcceptedConnection: Name: %s, TimeStamp: %s, %s"), *GetName(), FPlatformTime::StrTimestamp(), *Connection->Describe() );
	NETWORK_PROFILER( GNetworkProfiler.TrackEvent( TEXT( "OPEN" ), *( GetName() + TEXT( " " ) + Connection->LowLevelGetRemoteAddress() ), Connection ) );
}

同时NetDriver里也会收到通知，会加入到所有的客户端连接集合MappedClientConnections里，同通知ReplicationDriver在执行后续的Actor同步的时候考虑这个新连接:

void UNetDriver::AddClientConnection(UNetConnection* NewConnection)
{
	LLM_SCOPE_BYTAG(NetDriver);

	SCOPE_CYCLE_COUNTER(Stat_NetDriverAddClientConnection);

	UE_CLOG(!DDoS.CheckLogRestrictions(), LogNet, Log, TEXT("AddClientConnection: Added client connection: %s"), *NewConnection->Describe());

	ClientConnections.Add(NewConnection);

	TSharedPtr<const FInternetAddr> ConnAddr = NewConnection->GetRemoteAddr();

	if (ConnAddr.IsValid())
	{
		MappedClientConnections.Add(ConnAddr.ToSharedRef(), NewConnection);

		// On the off-chance of the same IP:Port being reused, check RecentlyDisconnectedClients
		int32 RecentDisconnectIdx = RecentlyDisconnectedClients.IndexOfByPredicate(
			[&ConnAddr](const FDisconnectedClient& CurElement)
			{
				return *ConnAddr == *CurElement.Address;
			});

		if (RecentDisconnectIdx != INDEX_NONE)
		{
			RecentlyDisconnectedClients.RemoveAt(RecentDisconnectIdx);
		}
	}

	if (ReplicationDriver)
	{
		ReplicationDriver->AddClientConnection(NewConnection);
	}
	// 省略一些代码
}

至此在UE的服务端，这个客户端连接建立的流程就完整走完了。

前面说过在StatelessConnectHandlerComponent验证完Challenge之后会发送一个ACK包到客户端，客户端处理这个包的代码仍然是StatelessConnectHandlerComponent::Incoming,只不过分支不一样:

void StatelessConnectHandlerComponent::Incoming(FBitReader& Packet)
{
		bool bHandshakePacket = !!Packet.ReadBit() && !Packet.IsError();

	if (bHandshakePacket)
	{
		FParsedHandshakeData HandshakeData;

		bHandshakePacket = ParseHandshakePacket(Packet, HandshakeData);

		if (bHandshakePacket)
		{
			const bool bIsChallengePacket = HandshakeData.HandshakePacketType == EHandshakePacketType::Challenge && HandshakeData.Timestamp > 0.0;
			const bool bIsInitialChallengePacket = bIsChallengePacket && State != UE::Handler::Component::State::Initialized;
			const bool bIsUpgradePacket = HandshakeData.HandshakePacketType == EHandshakePacketType::VersionUpgrade;

			if (Handler->Mode == UE::Handler::Mode::Client && bHasValidClientID && (bHasValidSessionID || bIsInitialChallengePacket || bIsUpgradePacket))
			{
				if (State == UE::Handler::Component::State::UnInitialized || State == UE::Handler::Component::State::InitializedOnLocal)
				{
					if (HandshakeData.bRestartHandshake)
					{
#if !UE_BUILD_SHIPPING
						UE_LOG(LogHandshake, Log, TEXT("Ignoring restart handshake request, while already restarted."));
#endif
					}
					// Receiving challenge
					else if (bIsChallengePacket)
					{
						// 忽略已经介绍过的部分代码
					}
					// Receiving challenge ack, verify the timestamp is < 0.0f
					else if (HandshakeData.HandshakePacketType == EHandshakePacketType::Ack && HandshakeData.Timestamp < 0.0)
					{
						if (!bRestartedHandshake)
						{
							UNetConnection* ServerConn = (Driver != nullptr ? ToRawPtr(Driver->ServerConnection) : nullptr);

							// Extract the initial packet sequence from the random Cookie data
							if (ensure(ServerConn != nullptr))
							{
								int16* CurSequence = (int16*)HandshakeData.Cookie;

								int32 ServerSequence = *CurSequence & (MAX_PACKETID - 1);
								int32 ClientSequence = *(CurSequence + 1) & (MAX_PACKETID - 1);

								ServerConn->InitSequence(ServerSequence, ClientSequence);
							}

							// Save the final authorized cookie
							FMemory::Memcpy(AuthorisedCookie, HandshakeData.Cookie, UE_ARRAY_COUNT(AuthorisedCookie));
						}

						// Now finish initializing the handler - flushing the queued packet buffer in the process.
						SetState(UE::Handler::Component::State::Initialized);
						Initialized();

						bRestartedHandshake = false;

						// Reset packet count clientside, due to how it affects protocol version fallback selection
						SentHandshakePacketCount = 0;
					}
				}
			}
		}
	}
}

在这里又会通过Cookie重新解析出下行包的初始序列号ServerSequence和上行包的初始序列号ClientSequence，并用这两个序列号来初始化服务端连接。同时这个完整的Cookie会被拷贝到AuthorisedCookie，以方便后面的断线重连来使用。

玩家登录

后续的Initialized函数调用会最终触发到PacketHandler::HandlerInitialized,在这个函数的末尾会调用创建PacketHandler的时候传入来的HandshakeCompleteDel:

void PacketHandler::HandlerInitialized()
{
	// 省略一些代码
	SetState(UE::Handler::State::Initialized);

	if (bBeganHandshaking)
	{
		HandshakeCompleteDel.ExecuteIfBound();
	}
}

void UPendingNetGame::BeginHandshake()
{
	// Kick off the connection handshake
	UNetConnection* ServerConn = NetDriver->ServerConnection;
	if (ServerConn->Handler.IsValid())
	{
		ServerConn->Handler->BeginHandshaking(
			FPacketHandlerHandshakeComplete::CreateUObject(this, &UPendingNetGame::SendInitialJoin));
	}
	else
	{
		SendInitialJoin();
	}
}

而这个HandshakeCompleteDel在UPendingNetGame::BeginHandshake的时候会设置为SendInitialJoin，在这个函数里会发送一个NMT_Hello消息到服务器端，这个消息里会带上可能的加密密钥EncryptionToken:

void UPendingNetGame::SendInitialJoin()
{
	if (NetDriver != nullptr)
	{
		UNetConnection* ServerConn = NetDriver->ServerConnection;

		if (ServerConn != nullptr)
		{
			uint8 IsLittleEndian = uint8(PLATFORM_LITTLE_ENDIAN);
			check(IsLittleEndian == !!IsLittleEndian); // should only be one or zero

			const int32 AllowEncryption = CVarNetAllowEncryption.GetValueOnGameThread();
			FString EncryptionToken;

			if (AllowEncryption != 0)
			{
				EncryptionToken = URL.GetOption(TEXT("EncryptionToken="), TEXT(""));
			}

			bool bEncryptionRequirementsFailure = false;

			// 忽略加密检查相关代码
			
			if (!bEncryptionRequirementsFailure)
			{
				uint32 LocalNetworkVersion = FNetworkVersion::GetLocalNetworkVersion();

				UE_LOG(LogNet, Log, TEXT("UPendingNetGame::SendInitialJoin: Sending hello. %s"), *ServerConn->Describe());

				EEngineNetworkRuntimeFeatures LocalNetworkFeatures = NetDriver->GetNetworkRuntimeFeatures();
				FNetControlMessage<NMT_Hello>::Send(ServerConn, IsLittleEndian, LocalNetworkVersion, EncryptionToken, LocalNetworkFeatures);


				ServerConn->FlushNet();
			}
			else
			{
				UE_LOG(LogNet, Error, TEXT("UPendingNetGame::SendInitialJoin: EncryptionToken is empty when 'net.AllowEncryption' requires it."));

				ConnectionError = TEXT("EncryptionToken not set.");
			}
		}
	}
}

这个FNetControlMessage会发送一个ControlMessage通过UControllChannel到对端，发送接口调用的是UChannel::SendBunch函数，而不是之前提到的LowLevelSend。通过UControllChannel发送的消息将会带上bReliable标记，如果消息丢失会执行重传。

UE针对接收到的UControllChannel数据会有专门的函数void UControlChannel::ReceivedBunch( FInBunch& Bunch )去处理，对于NMT_Hello这个消息会通过NetDriver上的Notify对象进行转发:

if (Connection->Driver->Notify != nullptr)
{
	// Process control message on client/server connection
	Connection->Driver->Notify->NotifyControlMessage(Connection, MessageType, Bunch);
}

在服务端这个Notify对象就是UWorld，在客户端这个Notify对象就是UPendingNetGame，所以服务端这里会调用到UWorld::NotifyControlMessage:

void UWorld::NotifyControlMessage(UNetConnection* Connection, uint8 MessageType, class FInBunch& Bunch)
{
	if( NetDriver->ServerConnection )
	{
		// 客户端代码 先忽略
	}
	else
	{
		// We are the server.
#if !(UE_BUILD_SHIPPING || UE_BUILD_TEST)
		UE_LOG(LogNet, Verbose, TEXT("Level server received: %s"), FNetControlMessageInfo::GetName(MessageType));
#endif
		if ( !Connection->IsClientMsgTypeValid( MessageType ) )
		{
			// If we get here, either code is mismatched on the client side, or someone could be spoofing the client address
			UE_LOG(LogNet, Error, TEXT( "IsClientMsgTypeValid FAILED (%i): Remote Address = %s" ), (int)MessageType,
					ToCStr(Connection->LowLevelGetRemoteAddress(true)));
			Bunch.SetError();
			return;
		}
		
		switch (MessageType)
		{
			// 暂时省略每个类型的消息处理
		}
	}
}

这里的NotifyControlMessage会根据传入的MessageType来执行各自的逻辑，对于NMT_Hello来说，会首先利用FNetControlMessage来反序列化出来传入的各种字段，检查完网络兼容性和加密Token之后，会通过SendChallengeControlMessage予以回包:

case NMT_Hello:
{
	uint8 IsLittleEndian = 0;
	uint32 RemoteNetworkVersion = 0;
	uint32 LocalNetworkVersion = FNetworkVersion::GetLocalNetworkVersion();
	FString EncryptionToken;

	EEngineNetworkRuntimeFeatures LocalNetworkFeatures = NetDriver->GetNetworkRuntimeFeatures();
	EEngineNetworkRuntimeFeatures RemoteNetworkFeatures = EEngineNetworkRuntimeFeatures::None;

	if (FNetControlMessage<NMT_Hello>::Receive(Bunch, IsLittleEndian, RemoteNetworkVersion, EncryptionToken, RemoteNetworkFeatures))
	{
		const bool bIsNetCLCompatible = FNetworkVersion::IsNetworkCompatible(LocalNetworkVersion, RemoteNetworkVersion);
		const bool bAreNetFeaturesCompatible = FNetworkVersion::AreNetworkRuntimeFeaturesCompatible(LocalNetworkFeatures, RemoteNetworkFeatures);

		if (!bIsNetCLCompatible || !bAreNetFeaturesCompatible)
		{
			// 忽略网络格式不匹配的处理
		}
		else
		{
			if (EncryptionToken.IsEmpty())
			{
				EEncryptionFailureAction FailureResult = EEncryptionFailureAction::Default;
				
				if (FNetDelegates::OnReceivedNetworkEncryptionFailure.IsBound())
				{
					FailureResult = FNetDelegates::OnReceivedNetworkEncryptionFailure.Execute(Connection);
				}

				const bool bGameplayDisableEncryptionCheck = FailureResult == EEncryptionFailureAction::AllowConnection;
				const bool bEncryptionRequired = NetDriver->IsEncryptionRequired() && !bGameplayDisableEncryptionCheck;

				if (!bEncryptionRequired)
				{
					Connection->SendChallengeControlMessage();
				}
				else
				{
					// 忽略强制要求加密的错误处理
				}
			}
		}
	}
}

这个SendChallengeControlMessage又是发送一个FNetControlMessage<NMT_Challenge>的包到客户端，这个包里唯一的参数就是当前服务器的时间:

void UNetConnection::SendChallengeControlMessage()
{
	if (GetConnectionState() != USOCK_Invalid && GetConnectionState() != USOCK_Closed && Driver)
	{
		Challenge = FString::Printf(TEXT("%08X"), FPlatformTime::Cycles());
		SetExpectedClientLoginMsgType(NMT_Login);
		FNetControlMessage<NMT_Challenge>::Send(this, Challenge);
		FlushNet();
	}
	else
	{
		UE_LOG(LogNet, Log, TEXT("UWorld::SendChallengeControlMessage: connection in invalid state. %s"), *Describe());
	}
}

客户端接收这个NMT_Challenge的地方在UPendingNetGame::NotifyControlMessage,解析完Challenge之后，又会构造一个FNetControlMessage<NMT_Login>消息发送到服务端，这个消息里会带上当前客户端玩家的唯一ID、用户名以及一些登录相关的额外信息:

case NMT_Challenge:
{
	// Challenged by server.
	if (FNetControlMessage<NMT_Challenge>::Receive(Bunch, Connection->Challenge))
	{
		FURL PartialURL(URL);
		PartialURL.Host = TEXT("");
		PartialURL.Port = PartialURL.UrlConfig.DefaultPort; // HACK: Need to fix URL parsing 
		PartialURL.Map = TEXT("");

		for (int32 i = URL.Op.Num() - 1; i >= 0; i--)
		{
			if (URL.Op[i].Left(5) == TEXT("game="))
			{
				URL.Op.RemoveAt(i);
			}
		}

		ULocalPlayer* LocalPlayer = GEngine->GetFirstGamePlayer(this);
		if (LocalPlayer)
		{
			// Send the player nickname if available
			FString OverrideName = LocalPlayer->GetNickname();
			if (OverrideName.Len() > 0)
			{
				PartialURL.AddOption(*FString::Printf(TEXT("Name=%s"), *OverrideName));
			}

			// Send any game-specific url options for this player
			FString GameUrlOptions = LocalPlayer->GetGameLoginOptions();
			if (GameUrlOptions.Len() > 0)
			{
				PartialURL.AddOption(*FString::Printf(TEXT("%s"), *GameUrlOptions));
			}

			// Send the player unique Id at login
			Connection->PlayerId = LocalPlayer->GetPreferredUniqueNetId();
		}

		// Send the player's online platform name
		FName OnlinePlatformName = NAME_None;
		if (const FWorldContext* const WorldContext = GEngine->GetWorldContextFromPendingNetGame(this))
		{
			if (WorldContext->OwningGameInstance)
			{
				OnlinePlatformName = WorldContext->OwningGameInstance->GetOnlinePlatformName();
			}
		}

		Connection->ClientResponse = TEXT("0");
		FString URLString(PartialURL.ToString());
		FString OnlinePlatformNameString = OnlinePlatformName.ToString();

		FNetControlMessage<NMT_Login>::Send(Connection, Connection->ClientResponse, URLString, Connection->PlayerId, OnlinePlatformNameString);
		NetDriver->ServerConnection->FlushNet();
	}
	else
	{
		Connection->Challenge.Empty();
	}

	break;
}

当服务端的UWorld收到这个NMT_Login消息之后,先解析出传入参数，然后通知GameMode来检查这个玩家是否被允许登录，如果允许则会调用到UWorld::PreLoginComplete来准备登录:

case NMT_Login:
{
	// Admit or deny the player here.
	FUniqueNetIdRepl UniqueIdRepl;
	FString OnlinePlatformName;
	FString& RequestURL = Connection->RequestURL;

	// Expand the maximum string serialization size, to accommodate extremely large Fortnite join URL's.
	Bunch.ArMaxSerializeSize += (16 * 1024 * 1024);

	bool bReceived = FNetControlMessage<NMT_Login>::Receive(Bunch, Connection->ClientResponse, RequestURL, UniqueIdRepl,
															OnlinePlatformName);

	Bunch.ArMaxSerializeSize -= (16 * 1024 * 1024);

	if (bReceived)
	{
		// Only the options/portal for the URL should be used during join
		const TCHAR* NewRequestURL = *RequestURL;

		for (; *NewRequestURL != '\0' && *NewRequestURL != '?' && *NewRequestURL != '#'; NewRequestURL++){}


		UE_LOG(LogNet, Log, TEXT("Login request: %s userId: %s platform: %s"), NewRequestURL, UniqueIdRepl.IsValid() ? *UniqueIdRepl.ToDebugString() : TEXT("UNKNOWN"), *OnlinePlatformName);

		// Compromise for passing splitscreen playercount through to gameplay login code,
		// without adding a lot of extra unnecessary complexity throughout the login code.
		// NOTE: This code differs from NMT_JoinSplit, by counting + 1 for SplitscreenCount
		//			(since this is the primary connection, not counted in Children)
		FURL InURL( NULL, NewRequestURL, TRAVEL_Absolute );

		if ( !InURL.Valid )
		{
			RequestURL = NewRequestURL;

			UE_LOG( LogNet, Error, TEXT( "NMT_Login: Invalid URL %s" ), *RequestURL );
			Bunch.SetError();
			break;
		}

		int32 SplitscreenCount = FMath::Min(Connection->Children.Num() + 1, 255);

		// Don't allow clients to specify this value
		InURL.RemoveOption(TEXT("SplitscreenCount"));
		InURL.AddOption(*FString::Printf(TEXT("SplitscreenCount=%i"), SplitscreenCount));

		RequestURL = InURL.ToString();

		// skip to the first option in the URL
		const TCHAR* Tmp = *RequestURL;
		for (; *Tmp && *Tmp != '?'; Tmp++);

		// keep track of net id for player associated with remote connection
		Connection->PlayerId = UniqueIdRepl;

		// keep track of the online platform the player associated with this connection is using.
		Connection->SetPlayerOnlinePlatformName(FName(*OnlinePlatformName));

		// ask the game code if this player can join
		AGameModeBase* GameMode = GetAuthGameMode();
		AGameModeBase::FOnPreLoginCompleteDelegate OnComplete = AGameModeBase::FOnPreLoginCompleteDelegate::CreateUObject(
			this, &UWorld::PreLoginComplete, TWeakObjectPtr<UNetConnection>(Connection));
		if (GameMode)
		{
			GameMode->PreLoginAsync(Tmp, Connection->LowLevelGetRemoteAddress(), Connection->PlayerId, OnComplete);
		}
		else
		{
			OnComplete.ExecuteIfBound(FString());
		}
	}
	else
	{
		Connection->ClientResponse.Empty();
		RequestURL.Empty();
	}

	break;
}

由于GameMode验证登录是一个异步操作，所以这里使用的是连接对象弱指针，如果允许登录，则会执行WelcomePlayer操作:

void UWorld::PreLoginComplete(const FString& ErrorMsg, TWeakObjectPtr<UNetConnection> WeakConnection)
{
	UNetConnection* Connection = WeakConnection.Get();
	if (!PreLoginCheckError(Connection, ErrorMsg))
	{
		return;
	}

	WelcomePlayer(Connection);
}

在WelComePlayer内部会发送一个NMT_Welcome的控制消息到客户端，参数里会填充好当前服务器的LevelName,GameName，以通知客户端去加载指定的地图:

void UWorld::WelcomePlayer(UNetConnection* Connection)
{
#if !WITH_EDITORONLY_DATA
	ULevel* CurrentLevel = PersistentLevel;
#endif

	check(CurrentLevel);

	FString LevelName;

	const FSeamlessTravelHandler& SeamlessTravelHandler = GEngine->SeamlessTravelHandlerForWorld(this);
	if (SeamlessTravelHandler.IsInTransition())
	{
		// Tell the client to go to the destination map
		LevelName = SeamlessTravelHandler.GetDestinationMapName();
		Connection->SetClientWorldPackageName(NAME_None);
	}
	else
	{
		LevelName = CurrentLevel->GetOutermost()->GetName();
		Connection->SetClientWorldPackageName(CurrentLevel->GetOutermost()->GetFName());
	}
	if (UGameInstance* GameInst = GetGameInstance())
	{
		GameInst->ModifyClientTravelLevelURL(LevelName);
	}

	FString GameName;
	FString RedirectURL;
	if (AuthorityGameMode != NULL)
	{
		GameName = AuthorityGameMode->GetClass()->GetPathName();
		AuthorityGameMode->GameWelcomePlayer(Connection, RedirectURL);
	}

	FNetControlMessage<NMT_Welcome>::Send(Connection, LevelName, GameName, RedirectURL);

	// 忽略一些代码


	Connection->FlushNet();
	// don't count initial join data for netspeed throttling
	// as it's unnecessary, since connection won't be fully open until it all gets received, and this prevents later gameplay data from being delayed to "catch up"
	Connection->QueuedBits = 0;
	Connection->SetClientLoginState( EClientLoginState::Welcomed );		// Client has been told to load the map, will respond via SendJoin
}

客户端收到这个消息的额时候并不会立即就加载地图，而是将地图信息填充到UPendingGame::URL里,最后再给服务器发送一个NMT_Netspeed控制消息来通知当前的网速限制:

case NMT_Welcome:
{
	// Server accepted connection.
	FString GameName;
	FString RedirectURL;

	if (FNetControlMessage<NMT_Welcome>::Receive(Bunch, URL.Map, GameName, RedirectURL))
	{
		//GEngine->NetworkRemapPath(this, URL.Map);

		UE_LOG(LogNet, Log, TEXT("Welcomed by server (Level: %s, Game: %s)"), *URL.Map, *GameName);

		// extract map name and options
		{
			FURL DefaultURL;
			FURL TempURL(&DefaultURL, *URL.Map, TRAVEL_Partial);
			URL.Map = TempURL.Map;
			URL.RedirectURL = RedirectURL;
			URL.Op.Append(TempURL.Op);
		}

		if (GameName.Len() > 0)
		{
			URL.AddOption(*FString::Printf(TEXT("game=%s"), *GameName));
		}

		// Send out netspeed now that we're connected
		FNetControlMessage<NMT_Netspeed>::Send(Connection, Connection->CurrentNetSpeed);

		// We have successfully connected
		// TickWorldTravel will load the map and call LoadMapCompleted which eventually calls SendJoin
		bSuccessfullyConnected = true;
	}
	else
	{
		URL.Map.Empty();
	}

	break;
}

客户端地图加载的逻辑则是在UEngine::TickWorldTravel里，这个函数会检查PendingGame的URL是否被设置了，

void UEngine::TickWorldTravel(FWorldContext& Context, float DeltaSeconds)
{
	// Handle seamless traveling
	if (Context.SeamlessTravelHandler.IsInTransition())
	{
		// Note: SeamlessTravelHandler.Tick may automatically update Context.World and GWorld internally
		Context.SeamlessTravelHandler.Tick();
	}

	// 忽略服务端加载地图的逻辑

	// Handle client traveling.
	// 忽略一些其他分支的代码

	if( Context.PendingNetGame )
	{
		Context.PendingNetGame->Tick( DeltaSeconds );
		if ( Context.PendingNetGame && Context.PendingNetGame->ConnectionError.Len() > 0 )
		{
			BroadcastNetworkFailure(NULL, Context.PendingNetGame->NetDriver, ENetworkFailure::PendingConnectionFailure, Context.PendingNetGame->ConnectionError);
			CancelPending(Context);
		}
		else if (Context.PendingNetGame && Context.PendingNetGame->bSuccessfullyConnected && !Context.PendingNetGame->bSentJoinRequest && !Context.PendingNetGame->bLoadedMapSuccessfully && (Context.OwningGameInstance == NULL || !Context.OwningGameInstance->DelayPendingNetGameTravel()))
		{
			if (Context.PendingNetGame->HasFailedTravel())
			{
				BrowseToDefaultMap(Context);
				BroadcastTravelFailure(Context.World(), ETravelFailure::TravelFailure, TEXT("Travel failed for unknown reason"));
			}
			else if (!MakeSureMapNameIsValid(Context.PendingNetGame->URL.Map))
			{
				BrowseToDefaultMap(Context);
				BroadcastTravelFailure(Context.World(), ETravelFailure::PackageMissing, Context.PendingNetGame->URL.Map);
			}
			else if (!Context.PendingNetGame->bLoadedMapSuccessfully)
			{
				// Attempt to load the map.
				FString Error;

				const bool bLoadedMapSuccessfully = LoadMap(Context, Context.PendingNetGame->URL, Context.PendingNetGame, Error);

				if (Context.PendingNetGame != nullptr)
				{
					if (!Context.PendingNetGame->LoadMapCompleted(this, Context, bLoadedMapSuccessfully, Error))
					{
						BrowseToDefaultMap(Context);
						BroadcastTravelFailure(Context.World(), ETravelFailure::LoadMapFailure, Error);
					}
				}
				else
				{
					BrowseToDefaultMap(Context);
					BroadcastTravelFailure(Context.World(), ETravelFailure::TravelFailure, Error);
				}
			}
		}
		
		// 省略一些代码
	}
	else if (TransitionType == ETransitionType::WaitingToConnect)
	{
		TransitionType = ETransitionType::None;
	}

	return;
}

LoadMap内部会通过MovePendingLevel这个接口将当前NetDriver所绑定的Notify从PendingNetGame切换为当前的Uworld，这样后续的ControlChannel的消息回调都会通过UWorld来处理了，与服务器一样:

void UEngine::MovePendingLevel(FWorldContext &Context)
{
	check(Context.World());
	check(Context.PendingNetGame);

	Context.World()->SetNetDriver(Context.PendingNetGame->NetDriver);

	UNetDriver* NetDriver = Context.PendingNetGame->NetDriver;
	if (NetDriver)
	{
		// The pending net driver is renamed to the current "game net driver"
		NetDriver->SetNetDriverName(NAME_GameNetDriver);
		NetDriver->SetWorld(Context.World());

		FLevelCollection& SourceLevels = Context.World()->FindOrAddCollectionByType(ELevelCollectionType::DynamicSourceLevels);
		SourceLevels.SetNetDriver(NetDriver);

		if (FLevelCollection* StaticLevels = Context.World()->FindCollectionByType(ELevelCollectionType::StaticLevels))
		{
			StaticLevels->SetNetDriver(NetDriver);
		}
	}

	// Attach the DemoNetDriver to the world if there is one
	if (UDemoNetDriver* DemoNetDriver = Context.PendingNetGame->GetDemoNetDriver())
	{
		DemoNetDriver->SetWorld(Context.World());
		Context.World()->SetDemoNetDriver(DemoNetDriver);

		FLevelCollection& MainLevels = Context.World()->FindOrAddCollectionByType(ELevelCollectionType::DynamicSourceLevels);
		MainLevels.SetDemoNetDriver(DemoNetDriver);
	}

	// Reset the Navigation System
	Context.World()->SetNavigationSystem(nullptr);
}

由于LoadMap是一个异步的过程，所以加载完成的检查依然需要在UEngine::TickWorldTravel里去做，当地图加载完成之后，PendingNetGame->TravelCompleted就会被调用到,在这个函数里会发送一个NMT_Join控制消息来通知服务器客户端已经加载完了地图，可以进入服务器地图了:

if (Context.PendingNetGame && Context.PendingNetGame->bLoadedMapSuccessfully && (Context.OwningGameInstance == NULL || !Context.OwningGameInstance->DelayCompletionOfPendingNetGameTravel()))
{
	if (!Context.PendingNetGame->HasFailedTravel() )
	{
		Context.PendingNetGame->TravelCompleted(this, Context);
		Context.PendingNetGame = nullptr;
	}
	else
	{
		CancelPending(Context);
		BrowseToDefaultMap(Context);
		BroadcastTravelFailure(Context.World(), ETravelFailure::LoadMapFailure, TEXT("Travel failed for unknown reason"));
	}
}

void UPendingNetGame::TravelCompleted(UEngine* Engine, FWorldContext& Context)
{
	// Show connecting message, cause precaching to occur.
	Engine->TransitionType = ETransitionType::Connecting;

	Engine->RedrawViewports(false);

	// Send join.
	Context.PendingNetGame->SendJoin();
	Context.PendingNetGame->NetDriver = NULL;

	UE_LOGSTATUS(Log, TEXT("Pending net game travel completed"));
}

void UPendingNetGame::SendJoin()
{
	bSentJoinRequest = true;

	FNetControlMessage<NMT_Join>::Send(NetDriver->ServerConnection);
	NetDriver->ServerConnection->FlushNet(true);
}

这里将Context.PendingNetGame设置为nullptr的目的就是彻底消除对PendingNetGame的引用，这样在后续的GC过程中可以回收这个对象。

当服务端的UWorld接收到这个NMT_Join控制消息之后，开始为这个客户端连接创建对应的PlayerController，同时利用这个PlayerController来调用ClientTravel来跳转到最终的关卡:

case NMT_Join:
{
	if (Connection->PlayerController == NULL)
	{
		// Spawn the player-actor for this network player.
		FString ErrorMsg;
		UE_LOG(LogNet, Log, TEXT("Join request: %s"), *Connection->RequestURL);

		FURL InURL( NULL, *Connection->RequestURL, TRAVEL_Absolute );

		if ( !InURL.Valid )
		{
			UE_LOG( LogNet, Error, TEXT( "NMT_Login: Invalid URL %s" ), *Connection->RequestURL );
			Bunch.SetError();
			break;
		}

		Connection->PlayerController = SpawnPlayActor( Connection, ROLE_AutonomousProxy, InURL, Connection->PlayerId, ErrorMsg );
		if (Connection->PlayerController == NULL)
		{
			// 忽略错误处理
		}
		else
		{
			// Successfully in game.
			UE_LOG(LogNet, Log, TEXT("Join succeeded: %s"), *Connection->PlayerController->PlayerState->GetPlayerName());
			NETWORK_PROFILER(GNetworkProfiler.TrackEvent(TEXT("JOIN"), *Connection->PlayerController->PlayerState->GetPlayerName(), Connection));

			Connection->SetClientLoginState(EClientLoginState::ReceivedJoin);

			// if we're in the middle of a transition or the client is in the wrong world, tell it to travel
			FString LevelName;
			FSeamlessTravelHandler &SeamlessTravelHandler = GEngine->SeamlessTravelHandlerForWorld( this );

			if (SeamlessTravelHandler.IsInTransition())
			{
				// tell the client to go to the destination map
				LevelName = SeamlessTravelHandler.GetDestinationMapName();
			}
			else if (!Connection->PlayerController->HasClientLoadedCurrentWorld())
			{
				// tell the client to go to our current map
				FString NewLevelName = GetOutermost()->GetName();
				UE_LOG(LogNet, Log, TEXT("Client joined but was sent to another level. Asking client to travel to: '%s'"), *NewLevelName);
				LevelName = NewLevelName;
			}
			if (LevelName != TEXT(""))
			{
				Connection->PlayerController->ClientTravel(LevelName, TRAVEL_Relative, true);
			}

			// @TODO FIXME - TEMP HACK? - clear queue on join
			Connection->QueuedBits = 0;
		}
	}
	break;
}

在这个UWorld::SpawnPlayActor里会通过GameMode::Login来创建指定类型的PlayerController对象，同时设置好对应的Role:

APlayerController* UWorld::SpawnPlayActor(UPlayer* NewPlayer, ENetRole RemoteRole, const FURL& InURL, const FUniqueNetIdRepl& UniqueId, FString& Error, uint8 InNetPlayerIndex)
{
	Error = TEXT("");

	// Make the option string.
	FString Options;
	for (int32 i = 0; i < InURL.Op.Num(); i++)
	{
		Options += TEXT('?');
		Options += InURL.Op[i];
	}

	if (AGameModeBase* const GameMode = GetAuthGameMode())
	{
		// Give the GameMode a chance to accept the login
		APlayerController* const NewPlayerController = GameMode->Login(NewPlayer, RemoteRole, *InURL.Portal, Options, UniqueId, Error);
		if (NewPlayerController == NULL)
		{
			UE_LOG(LogSpawn, Warning, TEXT("Login failed: %s"), *Error);
			return NULL;
		}

		UE_LOG(LogSpawn, Log, TEXT("%s got player %s [%s]"), *NewPlayerController->GetName(), *NewPlayer->GetName(), UniqueId.IsValid() ? *UniqueId->ToString() : TEXT("Invalid"));

		// Possess the newly-spawned player.
		NewPlayerController->NetPlayerIndex = InNetPlayerIndex;
		NewPlayerController->SetRole(ROLE_Authority);
		NewPlayerController->SetReplicates(RemoteRole != ROLE_None);
		if (RemoteRole == ROLE_AutonomousProxy)
		{
			NewPlayerController->SetAutonomousProxy(true);
		}
		NewPlayerController->SetPlayer(NewPlayer);
		GameMode->PostLogin(NewPlayerController);
		return NewPlayerController;
	}

	UE_LOG(LogSpawn, Warning, TEXT("Login failed: No game mode set."));
	return nullptr;
}

在最后的GameMode->PostLogin里，还会建立这个客户端玩家对应的Pawn对象,并尝试去开启当前的Match:

void AGameModeBase::PostLogin(APlayerController* NewPlayer)
{
	// Runs shared initialization that can happen during seamless travel as well

	GenericPlayerInitialization(NewPlayer);

	// Perform initialization that only happens on initially joining a server

	UWorld* World = GetWorld();

	NewPlayer->ClientCapBandwidth(NewPlayer->Player->CurrentNetSpeed);

	// 忽略观战相关的代码

	if (GameSession)
	{
		GameSession->PostLogin(NewPlayer);
	}

	DispatchPostLogin(NewPlayer);

	// Now that initialization is done, try to spawn the player's pawn and start match
	HandleStartingNewPlayer(NewPlayer);
}

void AGameMode::HandleStartingNewPlayer_Implementation(APlayerController* NewPlayer)
{
	// If players should start as spectators, leave them in the spectator state
	if (!bStartPlayersAsSpectators && !MustSpectate(NewPlayer))
	{
		// If match is in progress, start the player
		if (IsMatchInProgress() && PlayerCanRestart(NewPlayer))
		{
			RestartPlayer(NewPlayer);
		}
		// Check to see if we should start right away, avoids a one frame lag in single player games
		else if (GetMatchState() == MatchState::WaitingToStart)
		{
			// Check to see if we should start the match
			if (ReadyToStartMatch())
			{
				StartMatch();
			}
		}
	}
}

在StartMatch里会为每一个PlayerController分配一个出生点，并在这个出生点使用GameModeBase::DefaultPawnClass这个配置来创建初始的客户端对应的Pawn：

APawn* AGameModeBase::SpawnDefaultPawnFor_Implementation(AController* NewPlayer, AActor* StartSpot)
{
	// Don't allow pawn to be spawned with any pitch or roll
	FRotator StartRotation(ForceInit);
	StartRotation.Yaw = StartSpot->GetActorRotation().Yaw;
	FVector StartLocation = StartSpot->GetActorLocation();

	FTransform Transform = FTransform(StartRotation, StartLocation);
	return SpawnDefaultPawnAtTransform(NewPlayer, Transform);
}

APawn* AGameModeBase::SpawnDefaultPawnAtTransform_Implementation(AController* NewPlayer, const FTransform& SpawnTransform)
{
	FActorSpawnParameters SpawnInfo;
	SpawnInfo.Instigator = GetInstigator();
	SpawnInfo.ObjectFlags |= RF_Transient;	// We never want to save default player pawns into a map
	UClass* PawnClass = GetDefaultPawnClassForController(NewPlayer);
	APawn* ResultPawn = GetWorld()->SpawnActor<APawn>(PawnClass, SpawnTransform, SpawnInfo);
	if (!ResultPawn)
	{
		UE_LOG(LogGameMode, Warning, TEXT("SpawnDefaultPawnAtTransform: Couldn't spawn Pawn of type %s at %s"), *GetNameSafe(PawnClass), *SpawnTransform.ToHumanReadableString());
	}
	return ResultPawn;
}

对应的还有一个与玩家绑定的重要的类型PlayerState，会在PlayerController的初始化函数里创建:

void AController::InitPlayerState()
{
	if ( GetNetMode() != NM_Client )
	{
		UWorld* const World = GetWorld();
		const AGameModeBase* GameMode = World ? World->GetAuthGameMode() : NULL;

		// If the GameMode is null, this might be a network client that's trying to
		// record a replay. Try to use the default game mode in this case so that
		// we can still spawn a PlayerState.
		if (GameMode == NULL)
		{
			const AGameStateBase* const GameState = World ? World->GetGameState() : NULL;
			GameMode = GameState ? GameState->GetDefaultGameMode() : NULL;
		}

		if (GameMode != NULL)
		{
			FActorSpawnParameters SpawnInfo;
			SpawnInfo.Owner = this;
			SpawnInfo.Instigator = GetInstigator();
			SpawnInfo.SpawnCollisionHandlingOverride = ESpawnActorCollisionHandlingMethod::AlwaysSpawn;
			SpawnInfo.ObjectFlags |= RF_Transient;	// We never want player states to save into a map

			TSubclassOf<APlayerState> PlayerStateClassToSpawn = GameMode->PlayerStateClass;
			if (PlayerStateClassToSpawn.Get() == nullptr)
			{
				UE_LOG(LogPlayerController, Log, TEXT("AController::InitPlayerState: the PlayerStateClass of game mode %s is null, falling back to APlayerState."), *GameMode->GetName());
				PlayerStateClassToSpawn = APlayerState::StaticClass();
			}

			SetPlayerState(World->SpawnActor<APlayerState>(PlayerStateClassToSpawn, SpawnInfo));
	
			// force a default player name if necessary
			if (PlayerState && PlayerState->GetPlayerName().IsEmpty())
			{
				// don't call SetPlayerName() as that will broadcast entry messages but the GameMode hasn't had a chance
				// to potentially apply a player/bot name yet
				
				PlayerState->SetPlayerNameInternal(GameMode->DefaultPlayerName.ToString());
			}
		}
	}
}

至此，一个客户端玩家对应的PlayerController, PlayerState, PlayerPawn三个Actor都在登录成功之后创建出来了。

玩家下线

在UE里玩家下线有只有一个入口，就是客户端对应的UNetConnection的CleanUp函数。这个UNetConnection::CleanUp函数的调用时机有很多，包括但不限于：客户端主动登出、客户端进程退出、网络异常以及服务器主动断开等。

void UNetConnection::CleanUp()
{
	// Remove UChildConnection(s)
	for (int32 i = 0; i < Children.Num(); i++)
	{
		Children[i]->CleanUp();
	}
	Children.Empty();

	if ( State != USOCK_Closed )
	{
		UE_LOG( LogNet, Log, TEXT( "UNetConnection::Cleanup: Closing open connection. %s" ), *Describe() );
	}

	Close();

	if (Driver != nullptr)
	{
		// Remove from driver.
		if (Driver->ServerConnection)
		{
			check(Driver->ServerConnection == this);
			Driver->ServerConnection = NULL;
		}
		else
		{
			check(Driver->ServerConnection == NULL);
			Driver->RemoveClientConnection(this);
		}
	}

	// 省略一些关于netchannel清理的代码

	if (GIsRunning)
	{
		DestroyOwningActor();
	}

	CleanupDormantActorState();

	Handler.Reset(NULL);

	SetClientLoginState(EClientLoginState::CleanedUp);

	Driver = nullptr;
}

在这个UNetConnection::CleanUp函数里，会调用Close函数，这个函数里会往当前的ControlChannel也就是Channels[0]发送一个关闭包，这个关闭包会触发ControlChannel的Close函数，然后调用FlushNet来将所有未发送的包都发送出去，这样对端就知道当前连接已经不在可用了，:

void UNetConnection::Close()
{
	if (IsInternalAck())
	{
		SetReserveDestroyedChannels(false);
		SetIgnoreReservedChannels(false);
	}

	if (Driver != nullptr && State != USOCK_Closed)
	{
		if (Channels[0] != nullptr)
		{
			Channels[0]->Close(EChannelCloseReason::Destroyed);
		}
		State = USOCK_Closed;

		if ((Handler == nullptr || Handler->IsFullyInitialized()) && HasReceivedClientPacket())
		{
			FlushNet();
		}

		// 省略一些代码
	}

	LogCallLastTime		= 0;
	LogCallCount		= 0;
	LogSustainedCount	= 0;
}

在执行完成Close操作之后，接下来会调用DestroyOwningActor函数来销毁当前UNetConnection对应的Actor，这个Actor就是PlayerController：

void UNetConnection::DestroyOwningActor()
{
	if (OwningActor != nullptr)
	{
		// Cleanup/Destroy the connection actor & controller
		if (!OwningActor->HasAnyFlags(RF_BeginDestroyed | RF_FinishDestroyed))
		{
			// UNetConnection::CleanUp can be called from UNetDriver::FinishDestroyed that is called from GC.
			OwningActor->OnNetCleanup(this);
		}
		OwningActor = nullptr;
		PlayerController = nullptr;
	}
	else
	{
		if (ClientLoginState < EClientLoginState::ReceivedJoin)
		{
			UE_LOG(LogNet, Log, TEXT("UNetConnection::PendingConnectionLost. %s bPendingDestroy=%d "), *Describe(), bPendingDestroy);
			FGameDelegates::Get().GetPendingConnectionLostDelegate().Broadcast(PlayerId);
		}
	}
}

默认的Actor::OnNetCleanup的实现是空的，没有任何逻辑，只是为了方便子类重写。当前的APlayerController::OnNetCleanup函数会通过Destroy函数来强行销毁自己:

void APlayerController::OnNetCleanup(UNetConnection* Connection)
{
	UWorld* World = GetWorld();
	// destroy the PC that was waiting for a swap, if it exists
	if (World != NULL)
	{
		World->DestroySwappedPC(Connection);
	}

	check(UNetConnection::GNetConnectionBeingCleanedUp == NULL);
	UNetConnection::GNetConnectionBeingCleanedUp = Connection;
	//@note: if we ever implement support for splitscreen players leaving a match without the primary player leaving, we'll need to insert
	// a call to ClearOnlineDelegates() here so that PlayerController.ClearOnlineDelegates can use the correct ControllerId (which lives
	// in ULocalPlayer)
	Player = NULL;
	NetConnection = NULL;	
	Destroy( true );
	UNetConnection::GNetConnectionBeingCleanedUp = NULL;
}

PlayerController在销毁之后会触发AController::Destroyed函数，这个函数会通知当前的GameMode执行Logout函数,：

void AController::Destroyed()
{
	if (GetLocalRole() == ROLE_Authority && PlayerState != NULL)
	{
		// if we are a player, log out
		AGameModeBase* const GameMode = GetWorld()->GetAuthGameMode();
		if (GameMode)
		{
			GameMode->Logout(this);
		}

		CleanupPlayerState();
	}

	UnPossess();
	GetWorld()->RemoveController( this );
	Super::Destroyed();
}

这个AGameModeBase::Logout会执行这个玩家下线的事件广播FGameModeEvents::GameModeLogoutEvent，然后通知GameSession执行NotifyLogout函数，这个函数会通知所有玩家当前玩家下线了,并一路通知到当前的OnlineSubsystem::UnregisterPlayer：

void AGameModeBase::Logout(AController* Exiting)
{
	APlayerController* PC = Cast<APlayerController>(Exiting);
	if (PC != nullptr)
	{
		FGameModeEvents::GameModeLogoutEvent.Broadcast(this, Exiting);
		K2_OnLogout(Exiting);

		if (GameSession)
		{
			GameSession->NotifyLogout(PC);
		}
	}
}
void AGameSession::NotifyLogout(const APlayerController* PC)
{
	// Unregister the player from the online layer
	UnregisterPlayer(PC);
}
void AGameSession::UnregisterPlayer(const APlayerController* ExitingPlayer)
{
	if (GetNetMode() != NM_Standalone &&
		ExitingPlayer != NULL &&
		ExitingPlayer->PlayerState &&
		ExitingPlayer->PlayerState->GetUniqueId().IsValid())
	{
		UnregisterPlayer(ExitingPlayer->PlayerState->SessionName, ExitingPlayer->PlayerState->GetUniqueId());
	}
}
void AGameSession::UnregisterPlayer(FName InSessionName, const FUniqueNetIdRepl& UniqueId)
{
	UWorld* World = GetWorld();
	if (GetNetMode() != NM_Standalone &&
		UniqueId.IsValid() &&
		UniqueId->IsValid())
	{
		// Remove the player from the session
		UOnlineEngineInterface::Get()->UnregisterPlayer(World, InSessionName, *UniqueId);
	}
}

此外在AGameMode这个子类里还有额外的Logout逻辑,在这个子类的Logout重载里，会通过AddInactivePlayer函数构造当前PlayerState的一个副本，复制所有的属性字段，然后将这个副本添加到InactivePlayers数组中，这个数组会在后续的断线重连里使用：

void AGameMode::Logout( AController* Exiting )
{
	APlayerController* PC = Cast<APlayerController>(Exiting);
	if ( PC != nullptr )
	{
		RemovePlayerControllerFromPlayerCount(PC);
		AddInactivePlayer(PC->PlayerState, PC);
	}

	Super::Logout(Exiting);
}

void AGameMode::AddInactivePlayer(APlayerState* PlayerState, APlayerController* PC)
{
	check(PlayerState)
	UWorld* LocalWorld = GetWorld();
	// don't store if it's an old PlayerState from the previous level or if it's a spectator... or if we are shutting down
	if (!PlayerState->IsFromPreviousLevel() && !MustSpectate(PC) && !LocalWorld->bIsTearingDown)
	{
		APlayerState* const NewPlayerState = PlayerState->Duplicate();
		if (NewPlayerState)
		{
			// Side effect of Duplicate() adding PlayerState to PlayerArray (see APlayerState::PostInitializeComponents)
			GameState->RemovePlayerState(NewPlayerState);

			// make PlayerState inactive
			NewPlayerState->SetReplicates(false);

			// delete after some time
			NewPlayerState->SetLifeSpan(InactivePlayerStateLifeSpan);

			// On console, we have to check the unique net id as network address isn't valid
			const bool bIsConsole = !PLATFORM_DESKTOP;
			// Assume valid unique ids means comparison should be via this method
			const bool bHasValidUniqueId = NewPlayerState->GetUniqueId().IsValid();
			// Don't accidentally compare empty network addresses (already issue with two clients on same machine during development)
			const bool bHasValidNetworkAddress = !NewPlayerState->SavedNetworkAddress.IsEmpty();
			const bool bUseUniqueIdCheck = bIsConsole || bHasValidUniqueId;
			
			// make sure no duplicates
			// 省略一些容错代码
			InactivePlayerArray.Add(NewPlayerState);

			// 省略一些容错代码
		}
	}
}

这个PlayerState->Duplicate会调用到APlayerState::CopyProperties，这里会复制当前PlayerState的所有属性字段到新创建的PlayerState中,如果创建了子类，则需要重载这个函数来增加子类属性的复制：

void APlayerState::CopyProperties(APlayerState* PlayerState)
{
	PlayerState->SetScore(GetScore());
	PlayerState->SetPing(GetPing());
	PlayerState->ExactPing = ExactPing;
	PlayerState->SetPlayerId(GetPlayerId());
	PlayerState->SetUniqueId(GetUniqueId().GetUniqueNetId());
	PlayerState->SetPlayerNameInternal(GetPlayerName());
	PlayerState->SetStartTime(GetStartTime());
	PlayerState->SavedNetworkAddress = SavedNetworkAddress;
}

当然这个离线的PlayerState并不是永久保存在InactivePlayerArray中的，而是会在InactivePlayerStateLifeSpan时间之后被销毁，目前这个时间默认是300秒,也就是五分钟，同时这个InactivePlayerArray有最大容量MaxInactivePlayers限制,默认为16个:

AGameMode::AGameMode(const FObjectInitializer& ObjectInitializer)
	: Super(ObjectInitializer)
{
	bDelayedStart = false;

	// One-time initialization
	PrimaryActorTick.bCanEverTick = true;
	PrimaryActorTick.TickGroup = TG_PrePhysics;
	MatchState = MatchState::EnteringMap;
	EngineMessageClass = UEngineMessage::StaticClass();
	GameStateClass = AGameState::StaticClass();
	MinRespawnDelay = 1.0f;
	InactivePlayerStateLifeSpan = 300.f;
	MaxInactivePlayers = 16;
}

完成了GameMode->Logout之后，还会调用CleanupPlayerState函数来销毁当前玩家的PlayerState：

void AController::CleanupPlayerState()
{
	PlayerState->Destroy();
	PlayerState = NULL;
}

然后调用AController::UnPossess来取消控制当前玩家的Pawn：

void AController::UnPossess()
{
	APawn* CurrentPawn = GetPawn();

	// No need to notify if we don't have a pawn
	if (CurrentPawn == nullptr)
	{
		return;
	}

	OnUnPossess();

	// Notify only when pawn has been successfully unpossessed by the native class.
	APawn* NewPawn = GetPawn();
	if (NewPawn != CurrentPawn)
	{
		ReceiveUnPossess(CurrentPawn);
		OnNewPawn.Broadcast(NewPawn);
	}
}

void AController::OnUnPossess()
{
	// Should not be called when Pawn is null but since OnUnPossess could be overridden
	// the derived class could have already cleared the pawn and then call its base class.
	if ( Pawn != NULL )
	{
		Pawn->UnPossessed();
		SetPawn(NULL);
	}
}

值得注意的是APawn::UnPossessed内部并不会执行当前APawn的销毁，只是将当前APawn的Controller设置为nullptr，并通知当前GameInstance当前Pawn的Controller变化为nullptr：

void APawn::UnPossessed()
{
	AController* const OldController = Controller;

	ForceNetUpdate();

	SetPlayerState(nullptr);
	SetOwner(nullptr);
	Controller = nullptr;

	// Unregister input component if we created one
	DestroyPlayerInputComponent();

	// dispatch Blueprint event if necessary
	if (OldController)
	{
		ReceiveUnpossessed(OldController);
	}

	if (UGameInstance* GameInstance = GetGameInstance())
	{
		GameInstance->GetOnPawnControllerChanged().Broadcast(this, nullptr);
	}

	ConsumeMovementInputVector();
}

但是AController的子类APlayerController在Destroyed函数里会通过PawnLeavingGame来销毁当前玩家的APawn：

void APlayerController::Destroyed()
{
	if (GetPawn() != NULL)
	{
		// Handle players leaving the game
		if (Player == NULL && GetLocalRole() == ROLE_Authority)
		{
			PawnLeavingGame();
		}
		else
		{
			UnPossess();
		}
	}

	if (GetSpectatorPawn() != NULL)
	{
		DestroySpectatorPawn();
	}
	// 省略一些代码
}
void APlayerController::PawnLeavingGame()
{
	if (GetPawn() != NULL)
	{
		GetPawn()->Destroy();
		SetPawn(NULL);
	}
}

综上，如果客户端的连接关闭了，对应的APlayerController、APlayerState、APawn三个对象都会被强制销毁。如果想在断线之后保留APawn，则需要在子类里重载APlayerController::PawnLeavingGame。

	/** Clean up when a Pawn's player is leaving a game. Base implementation destroys the pawn. */
	virtual void PawnLeavingGame();

断线重连

当前UE实现的断线重连支持两种情况：

1. 一种是在UNetConnection没有关闭情况下的断线重连，此时常见于客户端的网络切换导致的ip:port变化
2. 一种是UNetConnection关闭了，但是玩家的PlayerState、APawn等对象还没有被销毁时的重新登录，此时常见客户端的重启以及客户端设备的更换

客户端重启之后的重新登录这种情况最为简单，当服务端接收到这个新客户端的登录请求时，会在AGameMode::PostLogin里利用FindInactivePlayer来检查当前新登录玩家的UniqueId或者PlayerName是否匹配上了有之前存储的已经断开连接的PlayerState，如果匹配上了，就会将这个新的UNetConnection绑定到之前的PlayerState、APawn等对象上：

void AGameMode::PostLogin( APlayerController* NewPlayer )
{
	UWorld* World = GetWorld();

	// 省略一些无关代码

	// save network address for re-associating with reconnecting player, after stripping out port number
	FString Address = NewPlayer->GetPlayerNetworkAddress();
	int32 pos = Address.Find(TEXT(":"), ESearchCase::CaseSensitive);
	NewPlayer->PlayerState->SavedNetworkAddress = (pos > 0) ? Address.Left(pos) : Address;

	// check if this player is reconnecting and already has PlayerState
	FindInactivePlayer(NewPlayer);

	Super::PostLogin(NewPlayer);
}

bool AGameMode::FindInactivePlayer(APlayerController* PC)
{
	check(PC && PC->PlayerState);
	// don't bother for spectators
	if (MustSpectate(PC))
	{
		return false;
	}

	// On console, we have to check the unique net id as network address isn't valid
	const bool bIsConsole = !PLATFORM_DESKTOP;
	// Assume valid unique ids means comparison should be via this method
	const bool bHasValidUniqueId = PC->PlayerState->GetUniqueId().IsValid();
	// Don't accidentally compare empty network addresses (already issue with two clients on same machine during development)
	const bool bHasValidNetworkAddress = !PC->PlayerState->SavedNetworkAddress.IsEmpty();
	const bool bUseUniqueIdCheck = bIsConsole || bHasValidUniqueId;

	const FString NewNetworkAddress = PC->PlayerState->SavedNetworkAddress;
	const FString NewName = PC->PlayerState->GetPlayerName();
	for (int32 i=0; i < InactivePlayerArray.Num(); i++)
	{
		APlayerState* CurrentPlayerState = InactivePlayerArray[i];
		if ( (CurrentPlayerState == nullptr) || CurrentPlayerState->IsPendingKill() )
		{
			InactivePlayerArray.RemoveAt(i,1);
			i--;
		}
		else if ((bUseUniqueIdCheck && (CurrentPlayerState->GetUniqueId() == PC->PlayerState->GetUniqueId())) ||
				 (!bUseUniqueIdCheck && bHasValidNetworkAddress && (FCString::Stricmp(*CurrentPlayerState->SavedNetworkAddress, *NewNetworkAddress) == 0) && (FCString::Stricmp(*CurrentPlayerState->GetPlayerName(), *NewName) == 0)))
		{
			// found it!
			APlayerState* OldPlayerState = PC->PlayerState;
			PC->PlayerState = CurrentPlayerState;
			PC->PlayerState->SetOwner(PC);
			PC->PlayerState->SetReplicates(true);
			PC->PlayerState->SetLifeSpan(0.0f);
			OverridePlayerState(PC, OldPlayerState);
			GameState->AddPlayerState(PC->PlayerState);
			InactivePlayerArray.RemoveAt(i, 1);
			OldPlayerState->SetIsInactive(true);
			// Set the uniqueId to nullptr so it will not kill the player's registration 
			// in UnregisterPlayerWithSession()
			OldPlayerState->SetUniqueId(nullptr);
			OldPlayerState->Destroy();
			PC->PlayerState->OnReactivated();
			return true;
		}
		
	}
	return false;
}

如果我们的APlayerController在PawnLeavingGame的时候没有销毁对应的APawn，那么在这里找到老的PlayerState之后，还需要自己做一下逻辑来执行重新Posses。

UNetConnection没有关闭情况下的断线重连则复杂一些，涉及到一个重新握手的过程。但是此时客户端与服务器都不知道当前客户端的ip:port发生了改变，仍然以之前的UNetConnection来进行通信。此时服务端发现这个新的数据包来自于一个未知的ip:port，因此会使用StatelessConnectHandlerComponent来处理这个包。当发现这个数据包并不是请求连接建立的握手包，此时会通过StatelessConnectHandlerComponent::SendRestartHandshakeRequest发送一个回包，提示客户端重新执行身份验证来绑定之前的UNetConnection:

void StatelessConnectHandlerComponent::IncomingConnectionless(FIncomingPacketRef PacketRef)
{
	FBitReader& Packet = PacketRef.Packet;
	const TSharedPtr<const FInternetAddr> Address = PacketRef.Address;

	if (MagicHeader.Num() > 0)
	{
		// Don't bother with the expense of verifying the magic header here.
		uint32 ReadMagic = 0;
		Packet.SerializeBits(&ReadMagic, MagicHeader.Num());
	}

	bool bHandshakePacket = !!Packet.ReadBit() && !Packet.IsError();

	LastChallengeSuccessAddress = nullptr;

	if (bHandshakePacket)
	{
		// 省略正常的代码
	}
#if !UE_BUILD_SHIPPING
	else if (Packet.IsError())
	{
		UE_LOG(LogHandshake, Log, TEXT("IncomingConnectionless: Error reading handshake bit from packet."));
	}
#endif
	// Late packets from recently disconnected clients may incorrectly trigger this code path, so detect and exclude those packets
	else if (!Packet.IsError() && !PacketRef.Traits.bFromRecentlyDisconnected)
	{
		// The packet was fine but not a handshake packet - an existing client might suddenly be communicating on a different address.
		// If we get them to resend their cookie, we can update the connection's info with their new address.
		SendRestartHandshakeRequest(Address);
	}
}

在SendRestartHandshakeRequest里会构造一个RestartPacket，这个RestartPacket开头会先填充MagicHeader，然后接着两个都是1的bit,表示这是一个重启握手包,然后通过Driver->LowLevelSend发送回这个客户端:

void StatelessConnectHandlerComponent::SendRestartHandshakeRequest(const TSharedPtr<const FInternetAddr> ClientAddress)
{
	if (Driver != nullptr)
	{
		FBitWriter RestartPacket(GetAdjustedSizeBits(RESTART_HANDSHAKE_PACKET_SIZE_BITS) + 1 /* Termination bit */);
		uint8 bHandshakePacket = 1;
		uint8 bRestartHandshake = 1;

		if (MagicHeader.Num() > 0)
		{
			RestartPacket.SerializeBits(MagicHeader.GetData(), MagicHeader.Num());
		}

		RestartPacket.WriteBit(bHandshakePacket);
		RestartPacket.WriteBit(bRestartHandshake);

		CapHandshakePacket(RestartPacket);

		
		// Disable PacketHandler parsing, and send the raw packet
		PacketHandler* ConnectionlessHandler = Driver->ConnectionlessHandler.Get();

		if (ConnectionlessHandler != nullptr)
		{
			ConnectionlessHandler->SetRawSend(true);
		}
		{
			if (Driver->IsNetResourceValid())
			{
				FOutPacketTraits Traits;

				Driver->LowLevelSend(ClientAddress, RestartPacket.GetData(), RestartPacket.GetNumBits(), Traits);
			}
		}


		if (ConnectionlessHandler != nullptr)
		{
			ConnectionlessHandler->SetRawSend(false);
		}
	}
	else
	{
#if !UE_BUILD_SHIPPING
		UE_LOG(LogHandshake, Error, TEXT("Tried to send restart handshake packet without a net driver."));
#endif
	}
}

当客户端接收到这个握手包的时候，发现这个bRestartHandshake为1，此时会认为这是一个重启握手包，然后会通过NotifyHandshakeBegin重新执行身份验证,此时内部的bRestartedHandshake字段会被设置为true:

void StatelessConnectHandlerComponent::Incoming(FBitReader& Packet)
{
	if (MagicHeader.Num() > 0)
	{
		// Don't bother with the expense of verifying the magic header here.
		uint32 ReadMagic = 0;
		Packet.SerializeBits(&ReadMagic, MagicHeader.Num());
	}

	bool bHandshakePacket = !!Packet.ReadBit() && !Packet.IsError();

	if (bHandshakePacket)
	{
		bool bRestartHandshake = false;
		uint8 SecretId = 0;
		double Timestamp = 1.;
		uint8 Cookie[COOKIE_BYTE_SIZE];
		uint8 OrigCookie[COOKIE_BYTE_SIZE];

		bHandshakePacket = ParseHandshakePacket(Packet, bRestartHandshake, SecretId, Timestamp, Cookie, OrigCookie);

		if (bHandshakePacket)
		{
			if (Handler->Mode == Handler::Mode::Client)
			{
				if (State == Handler::Component::State::UnInitialized || State == Handler::Component::State::InitializedOnLocal)
				{
					// 忽略正常分支的处理
				}
				else if (bRestartHandshake)
				{
					uint8 ZeroCookie[COOKIE_BYTE_SIZE] = {0};
					bool bValidAuthCookie = FMemory::Memcmp(AuthorisedCookie, ZeroCookie, COOKIE_BYTE_SIZE) != 0;

					// The server has requested us to restart the handshake process - this is because
					// it has received traffic from us on a different address than before.
					if (ensure(bValidAuthCookie))
					{
						bool bPassedDelayCheck = false;
						bool bPassedDualIPCheck = false;
						double CurrentTime = FPlatformTime::Seconds();;

						if (!bRestartedHandshake)
						{
							// 省略一些检查逻辑 内部会设置 bPassedDelayCheck, bPassedDualIPCheck
						}

						LastRestartPacketTimestamp = CurrentTime;

						
						if (!bRestartedHandshake && bPassedDelayCheck && bPassedDualIPCheck)
						{
							UE_LOG(LogHandshake, Log, TEXT("Beginning restart handshake process."));

							bRestartedHandshake = true;

							SetState(Handler::Component::State::UnInitialized);
							NotifyHandshakeBegin();
						}
					}
				}
			}
		}
	}
}

然后在NotifyHandshakeComplete中会构造一个新的握手包，此时bRestartHandshake对应的bit会被设置为1，然后往ServerConnection发送这个新的握手包:

void StatelessConnectHandlerComponent::NotifyHandshakeBegin()
{
	if (Handler->Mode == Handler::Mode::Client)
	{
		UNetConnection* ServerConn = (Driver != nullptr ? Driver->ServerConnection : nullptr);

		if (ServerConn != nullptr)
		{
			FBitWriter InitialPacket(GetAdjustedSizeBits(HANDSHAKE_PACKET_SIZE_BITS) + 1 /* Termination bit */);
			uint8 bHandshakePacket = 1;

			if (MagicHeader.Num() > 0)
			{
				InitialPacket.SerializeBits(MagicHeader.GetData(), MagicHeader.Num());
			}

			InitialPacket.WriteBit(bHandshakePacket);


			// In order to prevent DRDoS reflection amplification attacks, clients must pad the packet to match server packet size
			uint8 bRestartHandshake = bRestartedHandshake ? 1 : 0;
			uint8 SecretIdPad = 0;
			uint8 PacketSizeFiller[28];

			InitialPacket.WriteBit(bRestartHandshake);
			InitialPacket.WriteBit(SecretIdPad);

			FMemory::Memzero(PacketSizeFiller, UE_ARRAY_COUNT(PacketSizeFiller));
			InitialPacket.Serialize(PacketSizeFiller, UE_ARRAY_COUNT(PacketSizeFiller));



			CapHandshakePacket(InitialPacket);


			// Disable PacketHandler parsing, and send the raw packet
			Handler->SetRawSend(true);

			{
				if (ServerConn->Driver->IsNetResourceValid())
				{
					FOutPacketTraits Traits;

					ServerConn->LowLevelSend(InitialPacket.GetData(), InitialPacket.GetNumBits(), Traits);
				}
			}

			Handler->SetRawSend(false);

			LastClientSendTimestamp = FPlatformTime::Seconds();
		}
		else
		{
			UE_LOG(LogHandshake, Error, TEXT("Tried to send handshake connect packet without a server connection."));
		}
	}
}

然后当服务端收到这个新的握手包的时候，对应的处理函数依然是StatelessConnectHandlerComponent::IncomingConnectionless，但是此时会暂时忽视解析出来的bRestartHandshake，而是把这个当作一个初始握手包来看待，此时会利用SendConnectChallenge来构造一个新的Cookie往下发一个Chanllenge包:

void StatelessConnectHandlerComponent::IncomingConnectionless(FIncomingPacketRef PacketRef)
{
	FBitReader& Packet = PacketRef.Packet;
	const TSharedPtr<const FInternetAddr> Address = PacketRef.Address;

	if (MagicHeader.Num() > 0)
	{
		// Don't bother with the expense of verifying the magic header here.
		uint32 ReadMagic = 0;
		Packet.SerializeBits(&ReadMagic, MagicHeader.Num());
	}

	bool bHandshakePacket = !!Packet.ReadBit() && !Packet.IsError();

	LastChallengeSuccessAddress = nullptr;

	if (bHandshakePacket)
	{
		bool bRestartHandshake = false;
		uint8 SecretId = 0;
		double Timestamp = 1.0;
		uint8 Cookie[COOKIE_BYTE_SIZE];
		uint8 OrigCookie[COOKIE_BYTE_SIZE];

		bHandshakePacket = ParseHandshakePacket(Packet, bRestartHandshake, SecretId, Timestamp, Cookie, OrigCookie);

		if (bHandshakePacket)
		{
			if (Handler->Mode == Handler::Mode::Server)
			{
				const bool bInitialConnect = Timestamp == 0.0;

				if (bInitialConnect)
				{
					SendConnectChallenge(Address);
				}
				// 省略后续代码
			}
		}
	}
}

当客户端收到这个Challenge包的时候，会利用SendChallengeResponse来构造一个回包，不过此时发现当前自己已经记录了正在重连的状态，因此会额外附加之前已经商定的AuthorisedCookie:

void StatelessConnectHandlerComponent::SendChallengeResponse(uint8 InSecretId, double InTimestamp, uint8 InCookie[COOKIE_BYTE_SIZE])
{
	UNetConnection* ServerConn = (Driver != nullptr ? Driver->ServerConnection : nullptr);

	if (ServerConn != nullptr)
	{
		int32 RestartHandshakeResponseSize = RESTART_RESPONSE_SIZE_BITS;

#if RESTART_HANDSHAKE_DIAGNOSTICS && !DISABLE_SEND_HANDSHAKE_DIAGNOSTICS
		bool bEnableDiagnostics = bRestartedHandshake && !!CVarNetRestartHandshakeDiagnostics.GetValueOnAnyThread();

		RestartHandshakeResponseSize = bEnableDiagnostics ? RESTART_RESPONSE_DIAGNOSTICS_SIZE_BITS : RestartHandshakeResponseSize;
#endif

		const int32 BaseSize = GetAdjustedSizeBits(bRestartedHandshake ? RestartHandshakeResponseSize : HANDSHAKE_PACKET_SIZE_BITS);
		FBitWriter ResponsePacket(BaseSize + 1 /* Termination bit */);
		uint8 bHandshakePacket = 1;
		uint8 bRestartHandshake = (bRestartedHandshake ? 1 : 0);

		if (MagicHeader.Num() > 0)
		{
			ResponsePacket.SerializeBits(MagicHeader.GetData(), MagicHeader.Num());
		}

		ResponsePacket.WriteBit(bHandshakePacket);
		ResponsePacket.WriteBit(bRestartHandshake);
		ResponsePacket.WriteBit(InSecretId);

		ResponsePacket << InTimestamp;
		ResponsePacket.Serialize(InCookie, COOKIE_BYTE_SIZE);

		if (bRestartedHandshake)
		{
			ResponsePacket.Serialize(AuthorisedCookie, COOKIE_BYTE_SIZE);

#if RESTART_HANDSHAKE_DIAGNOSTICS && !DISABLE_SEND_HANDSHAKE_DIAGNOSTICS
			if (bEnableDiagnostics)
			{
				ResponsePacket << HandshakeDiagnostics;
			}
#endif
		}
	}

	// 省略后续代码
}

当服务端收到这个ChallengeResponse包的时候，会发现此时bRestartHandshake为1，因此会认为这是一个重启握手包，此时会利用客户端发送过来的老的OrigCookie来填充AuthorisedCookie,而不是用新Challenge时构造的Cookie:

if (bValidCookieLifetime && bValidSecretIdTimestamp)
{
	// Regenerate the cookie from the packet info, and see if the received cookie matches the regenerated one
	uint8 RegenCookie[COOKIE_BYTE_SIZE];

	GenerateCookie(Address, SecretId, Timestamp, RegenCookie);

	bChallengeSuccess = FMemory::Memcmp(Cookie, RegenCookie, COOKIE_BYTE_SIZE) == 0;

	if (bChallengeSuccess)
	{
		if (bRestartHandshake)
		{
			FMemory::Memcpy(AuthorisedCookie, OrigCookie, UE_ARRAY_COUNT(AuthorisedCookie));
		}
		else
		{
			int16* CurSequence = (int16*)Cookie;

			LastServerSequence = *CurSequence & (MAX_PACKETID - 1);
			LastClientSequence = *(CurSequence + 1) & (MAX_PACKETID - 1);

			FMemory::Memcpy(AuthorisedCookie, Cookie, UE_ARRAY_COUNT(AuthorisedCookie));
		}

		bRestartedHandshake = bRestartHandshake;
		LastChallengeSuccessAddress = Address->Clone();


		// Now ack the challenge response - the cookie is stored in AuthorisedCookie, to enable retries
		SendChallengeAck(Address, AuthorisedCookie);
	}
}

同时外层处理函数UIpNetDriver::ProcessConnectionlessPacket发现此时重新握手成功之后，会执行客户端地址与之前UNetConnection的重新绑定，重点就是更新MappedClientConnections这个映射表:

UNetConnection* UIpNetDriver::ProcessConnectionlessPacket(FReceivedPacketView& PacketRef, const FPacketBufferView& WorkingBuffer)
{
	UNetConnection* ReturnVal = nullptr;
	TSharedPtr<StatelessConnectHandlerComponent> StatelessConnect;
	const TSharedPtr<const FInternetAddr>& Address = PacketRef.Address;
	FString IncomingAddress = Address->ToString(true);
	bool bPassedChallenge = false;
	bool bRestartedHandshake = false;
	bool bIgnorePacket = true;

	if (ConnectionlessHandler.IsValid() && StatelessConnectComponent.IsValid())
	{
		StatelessConnect = StatelessConnectComponent.Pin();

		EIncomingResult Result = ConnectionlessHandler->IncomingConnectionless(PacketRef);

		if (Result == EIncomingResult::Success)
		{
			bPassedChallenge = StatelessConnect->HasPassedChallenge(Address, bRestartedHandshake);

			if (bPassedChallenge)
			{
				if (bRestartedHandshake)
				{
					UE_LOG(LogNet, Log, TEXT("Finding connection to update to new address: %s"), *IncomingAddress);

					TSharedPtr<StatelessConnectHandlerComponent> CurComp;
					UIpConnection* FoundConn = nullptr;

					for (UNetConnection* const CurConn : ClientConnections)
					{
						CurComp = CurConn != nullptr ? CurConn->StatelessConnectComponent.Pin() : nullptr;

						if (CurComp.IsValid() && StatelessConnect->DoesRestartedHandshakeMatch(*CurComp))
						{
							FoundConn = Cast<UIpConnection>(CurConn);
							break;
						}
					}

					if (FoundConn != nullptr)
					{
						UNetConnection* RemovedConn = nullptr;
						TSharedRef<FInternetAddr> RemoteAddrRef = FoundConn->RemoteAddr.ToSharedRef();

						verify(MappedClientConnections.RemoveAndCopyValue(RemoteAddrRef, RemovedConn) && RemovedConn == FoundConn);


						// @todo: There needs to be a proper/standardized copy API for this. Also in IpConnection.cpp
						bool bIsValid = false;

						const FString OldAddress = RemoteAddrRef->ToString(true);

						RemoteAddrRef->SetIp(*Address->ToString(false), bIsValid);
						RemoteAddrRef->SetPort(Address->GetPort());


						MappedClientConnections.Add(RemoteAddrRef, FoundConn);


						// Make sure we didn't just invalidate a RecentlyDisconnectedClients entry, with the same address
						int32 RecentDisconnectIdx = RecentlyDisconnectedClients.IndexOfByPredicate(
							[&RemoteAddrRef](const FDisconnectedClient& CurElement)
							{
								return *RemoteAddrRef == *CurElement.Address;
							});

						if (RecentDisconnectIdx != INDEX_NONE)
						{
							RecentlyDisconnectedClients.RemoveAt(RecentDisconnectIdx);
						}


						ReturnVal = FoundConn;

						// We shouldn't need to log IncomingAddress, as the UNetConnection should dump it with it's description.
						UE_LOG(LogNet, Log, TEXT("Updated IP address for connection. Connection = %s, Old Address = %s"), *FoundConn->Describe(), *OldAddress);
					}
					else
					{
						UE_LOG(LogNet, Log, TEXT("Failed to find an existing connection with a matching cookie. Restarted Handshake failed."));
					}
				}


				// 省略无关代码
			}
		}
	}
	// 省略其他分支的代码
}

当客户端收到这个ChallengeAck包的时候，就可以认为连接已经重建好了，可以继续利用之前的UNetConnection来发送消息了。整个过程可以简化为下面这个流程图:

Mosaic Game 的场景管理

场景space是游戏中玩家之间绝大部分玩法活动的逻辑承载空间，特别是玩家间的强实时互动。脱离了场景这个实体的话，游戏服务器的逻辑就只剩下服务service承载的聊天、好友之类的弱实时互动，这样就退化成为了普通互联网服务器的样子。因此强实时场景的存在是游戏服务器和互联网服务器之间最大的差异。在游戏服务器中，一般会有一个场景服务space_service来集中管理所有space的生命周期，同时管理玩家进出场景相关流程。

场景创建流程

游戏中的场景是多种多样的，策划一般会以一个场景编号space_no来方便的区分各个不同的场景。在不同的数据表中，通过使用相同的space_no来配置同一个场景的各种相关数据， 这个space_no就充当了场景配置索引的作用。同时游戏里可以为同一个蓝本的场景建立各自独立的实例，所以在场景管理中，一般会使用唯一的索引space_id来标识服务器内的单个具体场景实例。因此在创建场景实体space_entity的时候，需要提供space_no与space_id:

void Meta(rpc) request_create_space(const utility::rpc_msg& data,  std::uint32_t space_no, const std::string& space_id, const std::string& pref_game_id, const json::object_t& init_info);

由于这两个参数是外部传入的，所以有可能是非法的，例如不存在的space_no或者重复的space_id，所以这个函数的开头要做一堆的合法性检查：

if (!m_space_config_data)
{
	m_logger->error("fail to create space {} m_space_config_data null ", space_no);
	reply_msg.err = "m_space_config_data null";
	break;
}
cur_space_sysd = m_space_config_data->get_row(space_no);
if (!cur_space_sysd.valid())
{
	m_logger->error("fail to create space {} invalid space no ", space_no);
	reply_msg.err = "invalid space no";
	break;
}

if (!cur_space_sysd.expect_value(std::string("space_type"), cur_space_type_no))
{
	m_logger->error("fail to create space {} space_type empty ", space_no);
	reply_msg.err = "invalid space_no";
	break;		
}
cur_space_type_info = misc::space_type_info_mgr::get_space_type_info(cur_space_type_no);
if (!cur_space_type_info)
{
	m_logger->error("fail to create space {} space type info  empty  for space type {}", space_no, cur_space_type_no);
	reply_msg.err = "invalid space_no";
	break;				
}

if (!cur_space_sysd.expect_value(std::string("map_range"), map_range))
{
	m_logger->error("fail to create space {} map_range empty ", space_no);
	reply_msg.err = "invalid space_no";
	break;
}
if(space_id.empty())
{
	m_logger->error("fail to create space {} space_id empty ");
	reply_msg.err = "duplicated space_id";
	break;
}
if (m_space_types.find(space_id) != m_space_types.end())
{
	m_logger->error("fail to create space {} duplicated space id ", space_id);
	reply_msg.err = "duplicated space_id";
	break;
}

在space_service上使用了一个map来记录每个已经创建了space的space_id对应场景类型信息space_type_info:

struct space_type_info
{
	union
	{
		struct 
		{
			std::uint32_t is_union_space:1; //是否是大世界可分块场景
			std::uint32_t is_town_space:1; // 是否是城镇场景
			std::uint32_t is_player_dungeon:1; // 是否是单人副本
			std::uint32_t is_team_dungeon:1; //是否是组队副本
			std::uint32_t is_match_space:1; // 是否是匹配场景
			std::uint32_t auto_select_when_empty_id:1; // 空space_id进入时自动选择负载最低的instance
			std::uint32_t auto_create_new_heavy_load:1; // 高负载下自动创建新场景
			std::uint32_t support_back_return:1; // 是否支持离开后再回来
		};
		std::uint32_t all_flags = 0;
		
		
	};
	std::uint32_t space_type; // 场景类型
	std::uint32_t max_player_load; // 单场景最大玩家数量
};

std::unordered_map<std::string, const misc::space_type_info*> m_space_types;

这里并没有存储space_id到space_no的直接映射，因为所有场景相关接口里都会同时提供space_id,space_no这两个参数，校验这两个参数是否匹配可以通过space_service上的m_spaces_by_no字段来实现，这个字段存储了space_no到space_id集合的映射：

std::unordered_map<std::uint32_t, std::unordered_set<std::string>> m_spaces_by_no;

request_create_space里还有一个非常重要的参数pref_game_id，这个参数代表要将这个场景创建在哪一个进程上，如果没有指定的话，space_service会使用自己的负载均衡策略来选择一个合适的game_id来做填充：

if (dest_game_id.empty())
{
	dest_game_id = choose_game_for_space(space_no, cur_space_sysd);
	if (dest_game_id.empty())
	{
		m_logger->error("fail to choose game for space {}", space_no);
		reply_msg.err = "cant find allocate game_id";
		break;
	}
}
else
{
	auto cur_game_iter = m_game_loads.find(dest_game_id);
	if (cur_game_iter == m_game_loads.end())
	{
		m_logger->error("fail to find game {} to create space {}", dest_game_id, space_no);
		reply_msg.err = "invalid  game_id";
		break;
	}
}

目前的负载均衡策略非常的简陋，直接选取当前负载最低的进程作为结果返回，这里对于正在创建过程中的场景增加10的权重，以避免场景负载延迟上报的影响：

std::string space_service::choose_game_for_space(std::uint32_t space_no, typed_matrix::typed_row space_sysd)
{
	float temp_min_load = 10000.0f;
	std::string min_game_id;
	for (const auto &one_game_info : m_game_loads)
	{
		auto cur_game_load = one_game_info.second.cur_load + 10.0*one_game_info.second.creating_spaces.size();
		if (cur_game_load < temp_min_load)
		{
			min_game_id = one_game_info.first;
			temp_min_load = cur_game_load;
		}
	}
	return min_game_id;
}

在所有的合法性检查都通过之后，开始通知目标进程来创建这个场景：

std::string space_service::do_create_space(std::uint32_t cur_space_no, std::uint32_t cur_space_type, const std::string& pref_space_id,  const std::string& dest_game_id, const json::object_t &init_info)
{
	auto cur_space_sysd = m_space_config_data->get_row(cur_space_no);
	auto cur_game_id = dest_game_id;
	if(cur_game_id.empty())
	{
		choose_game_for_space(cur_space_no, cur_space_sysd);
	}
	
	auto cur_space_id = pref_space_id;
	if(pref_space_id.empty())
	{
		cur_space_id = get_server()->gen_unique_str();
	}

	m_logger->info("try create  space no {} with id {}", cur_space_no, cur_space_id);
	auto cur_space_type_info = misc::space_type_info_mgr::get_space_type_info(cur_space_type);
	std::string cell_space_id;
	std::string union_space_id;
	json::object_t space_init_info = init_info;
	utility::rpc_msg cur_msg;
	cur_msg.cmd = "notify_create_space";
	cur_msg.args.reserve(10);
	if(cur_space_type_info->is_union_space)
	{
		cell_space_id = get_server()->gen_unique_str();
		union_space_id = cur_space_id;
		std::array<std::array<double, 3>, 2> map_range;
		if(!cur_space_sysd.expect_value(std::string("map_range"), map_range))
		{
			return {};
		}
		utility::cell_region::cell_bound cur_map_range;
		cur_map_range.left_x = map_range[0][0];
		cur_map_range.right_x = map_range[1][0];
		cur_map_range.low_z = map_range[0][2];
		cur_map_range.high_z = map_range[1][2];
		std::unique_ptr<union_space_info> cur_union_space_ptr = std::make_unique<union_space_info>(union_space_id, cur_map_range, cur_space_no, cur_game_id, cell_space_id);
		space_init_info["components"]["cell"] = cur_union_space_ptr->cells.encode();
		m_union_spaces[union_space_id] = std::move(cur_union_space_ptr);
		m_space_types[union_space_id] = cur_space_type_info;
		m_logger->info("create cell {} for union space {}", cell_space_id, union_space_id);
		
	}
	else
	{
		cell_space_id = cur_space_id;
	}
	add_space_load_to_game(cur_space_no, cell_space_id, cur_game_id, union_space_id, cur_space_type_info);

	cur_msg.args.push_back(cell_space_id);
	cur_msg.args.push_back(cur_space_no);
	cur_msg.args.push_back(union_space_id);
	
	cur_msg.args.push_back(space_init_info);
	call_space_manager(cur_game_id, cur_msg);
	return cell_space_id;
}

这里的add_space_load_to_game会将这个场景加入到此game的正在创建场景集合creating_spaces中，作为一个临时占位的场景负载。因为进程的负载是定期采样的，如果在采样间隔内某个进程是负载最低的，负载均衡策略会将这一期间的所有场景都创建在同一进程上，从而导致此进程负载爆炸。所以为了避免出现短期内单一进程创建的场景太多，优化负载均衡：

void space_service::add_space_load_to_game(std::uint32_t space_no, const std::string &space_id, const std::string &game_id, const std::string &union_space_id, const misc::space_type_info *cur_space_type_info)
{
	if (union_space_id.empty())
	{
		std::unique_ptr<mono_space_info> cur_space_info = std::make_unique<mono_space_info>();
		cur_space_info->game_id = game_id;
		cur_space_info->space_id = space_id;
		cur_space_info->space_no = space_no;
		m_mono_spaces[space_id] = std::move(cur_space_info);
		m_game_loads[game_id].mono_spaces.insert(space_id);
		m_spaces_by_no[space_no].insert(space_id);
	}
	else
	{
		// 省略无关代码
	}

	m_space_types[space_id] = cur_space_type_info;

	m_game_loads[game_id].creating_spaces.insert(space_id);
}

在选择最佳game的时候会将这里creating_spaces所带来的预先负载设置为了常量10，无视了场景间的差异，其实更好的方法是在场景表中配置每个space_no对应的预先负载，这样就能更加精确的估算。

通知指定space_server进程创建新space的方式是往这个space_server上的space_manager发送创建场景的请求notify_create_space:

void space_service::call_space_manager(const std::string &game_id, const utility::rpc_msg &msg)
{
	get_server()->call_server(this, game_id + utility::rpc_anchor::seperator + "space_manager", msg);
}

这里的space_manager是每个space_server上都会存在的一个单例，space_server启动的时候就会自动初始化:

void manager_base::init_managers(space_server* in_space_server)
{
	offline_msg_manager::instance().init(in_space_server);
	email_manager::instance().init(in_space_server);
	notify_manager::instance().init(in_space_server);
	rank_manager::instance().init(in_space_server);
	space_manager::instance().init(in_space_server);
}

void space_server::do_start()
{
	entity::entity_manager::instance().init();
	
	json_stub::start();
	manager_base::init_managers(this);
	misc::stuff_utils::init();
	global_config_mgr::instance();
}

因此可以使用game_id + utility::rpc_anchor::seperator + "space_manager"的形式来拼接出对应的远程调用地址，因为在entity_manager没有处理这个rpc的情况下，会再往manager_base上尝试分发：

utility::rpc_msg::call_result manager_base::dispatch_rpc(const std::string& dest, const utility::rpc_msg& msg)
{
	auto temp_iter = m_managers.find(dest);
	if(temp_iter == m_managers.end())
	{
		return utility::rpc_msg::call_result::dest_not_found;
	}
	if(!temp_iter->second->support_rpc())
	{
		return utility::rpc_msg::call_result::rpc_not_found;
	}
	return temp_iter->second->rpc_owner_on_rpc(msg);
}

utility::rpc_msg::call_result space_server::on_server_rpc_msg(const std::string& dest, const utility::rpc_msg& cur_rpc_msg)
{
	auto dispatch_result = entity::entity_manager::instance().dispatch_rpc_msg(dest, cur_rpc_msg);
	if(dispatch_result != utility::rpc_msg::call_result::dest_not_found)
	{
		return dispatch_result;
	}
	return manager_base::dispatch_rpc(dest, cur_rpc_msg);

}

在space_manager接收到创建新场景的rpc请求之后，就会通过entity_manager来使用指定的参数来创建对应的space_entity：

void space_manager::notify_create_space(const utility::rpc_msg &data, const std::string &space_id, std::uint32_t space_no, const std::string& union_space_id, json::object_t &init_info)
{
	m_logger->info("notify_create_space space_id {} space_no {} union_space_id {} init_info {}", space_id, space_no, union_space_id, json(init_info).dump());
	std::string create_entity_error;
	init_info["space_no"] = space_no;
	init_info["union_space_id"] = union_space_id;
	auto cur_entity = m_server->create_entity("space_entity", space_id, m_server->gen_online_entity_id(), init_info, create_entity_error);
	if (!cur_entity)
	{
		m_logger->error("fail to create_space id {} with error {}", space_id, create_entity_error);
		return;
	}
	entity::space_entity *cur_space = dynamic_cast<entity::space_entity*>(cur_entity);
	m_spaces[space_id] = cur_space;
	report_space_created(space_id);
}

void space_manager::report_space_created(const std::string &space_id)
{
	utility::rpc_msg cur_msg;
	cur_msg.cmd = "report_space_created";
	cur_msg.args.push_back(m_server->local_stub_info().name);
	cur_msg.args.push_back(space_id);
	m_server->call_service( "space_service", cur_msg);
}

创建完成之后再通知space_service，可以走后续流程了。这个后续流程主要是将这个space的状态切换为ready，然后处理等待进入当前场景的所有玩家与队伍，逐个通知其可以进入：

void space_service::report_space_created(const utility::rpc_msg &data, const std::string &game_id, const std::string &space_id)
{

	auto cur_space_type_iter = m_space_types.find(space_id);
	if (cur_space_type_iter == m_space_types.end())
	{
		m_logger->error("cant find space {} report space created", space_id);
		return;
	}
	if (!cur_space_type_iter->second->is_union_space)
	{
		auto cur_space_iter = m_mono_spaces.find(space_id);
		if (cur_space_iter->second->game_id != game_id)
		{
			m_logger->error("space {} game id {} not match", space_id, game_id);
			return;
		}
		if (cur_space_iter->second->ready)
		{
			m_logger->error("space {} no {} game id {} already ready", space_id, cur_space_iter->second->space_no, game_id);
			return;
		}
		cur_space_iter->second->ready = true;
		for (const auto &one_pair : cur_space_iter->second->players)
		{
			// 之前在等待进入场景的玩家 现在重新开始进入场景
			utility::rpc_msg cur_msg;
			cur_msg.args.push_back(game_id);
			cur_msg.args.push_back(cur_space_iter->second->space_no);
			cur_msg.args.push_back(space_id);
			cur_msg.args.push_back(std::string());
			cur_msg.args.push_back(one_pair.second.enter_info);

			cur_msg.cmd = "reply_enter_space";

			get_server()->call_server(this, one_pair.second.call_anchor, cur_msg);
		}
		if (!cur_space_iter->second->team_id.empty())
		{
			utility::rpc_msg team_forward_msg;
			team_forward_msg.cmd = "notify_team_dungeon_created";
			team_forward_msg.set_args(cur_space_iter->second->team_id, cur_space_iter->second->space_no, space_id, game_id);
			get_server()->call_service("team_service", team_forward_msg);
		}
		
	}
	return;
}

了解了完整的场景创建流程之后，我们还需要明确场景的创建时机。常规的场景创建时机为按需创建，这种模式主要处理的是单人场景以及组队场景。每次玩家发起进入场景请求时，space_service会以这个场景编号space_no和场景实例space_id开启新场景实例的创建，同时将这个玩家记录在此创建中场景的等待进入玩家列表中。这个场景需要创建在游戏服务器另外的场景进程上，当这个场景创建完成之后会通知回space_service这个新场景实例已经可用，此时通知这个场景等待列表的玩家场景进入得到允许，可以迁入此场景。

在这种模式下，玩家进入一个场景的延迟会比较高，因为场景创建是一个比较消耗CPU的操作，需要加载很多资源和配置表格数据。特别是这个场景比较大，依赖的资源与数据非常多的情况下，玩家进入指定场景的延迟可能会有数秒。所以一般对于主城等大场景，采取的是预先创建的模式，在服务器启动的时候就创建好一定数量的常用场景，这样玩家进入这些场景的时候就可以避免巨大的等待延迟，达到秒切的目的。space_service::init中的代码就是为这个预先创建常用场景服务的，会遍历场景表里的每行配置数据，如果发现配置数据里开启了自动选择同编号随机场景auto_select_when_empty_id的功能，则会开启一个定时器来渐进的创建这些场景：

bool space_service::init(const json::object_t& data)
{
	if(!base_service::init(data))
	{
		return false;
	}
	auto cur_data_mgr = utility::typed_matrix_data_manager::instance();
	if(!cur_data_mgr)
	{
		return false;
	}
	m_space_config_data = cur_data_mgr->get("space");
	if (!m_space_config_data)
	{
		m_logger->error("cant get data config for space");
		return false;
	}
	auto space_no_column = m_space_config_data->get_column_idx("space_no");
	auto space_type_column = m_space_config_data->get_column_idx("space_type");
	if(!space_no_column.valid() || ! space_type_column.valid())
	{
		return false;
	}
	auto temp_row = m_space_config_data->begin_row();
	while(temp_row.valid())
	{
		auto cur_row = temp_row;
		temp_row = m_space_config_data->next_row(temp_row);
		std::uint32_t cur_space_no;
		std::uint32_t cur_space_type;
		if(!cur_row.expect_value(space_no_column, cur_space_no))
		{
			continue;
		}
		if(!cur_row.expect_value(space_type_column, cur_space_type))
		{
			continue;
		}
		auto cur_space_type_info = misc::space_type_info_mgr::get_space_type_info(cur_space_type);
		if(!cur_space_type_info)
		{
			continue;
		}
		if(cur_space_type_info->auto_select_when_empty_id)
		{
			m_init_spaces_to_create.push_back(std::make_pair(cur_space_no, cur_space_type));
		}
	}
	m_logger->info("init space is {}", serialize::encode(m_init_spaces_to_create).dump());
	add_timer_with_gap(std::chrono::milliseconds(5 * 1000), [=]()
	{
		create_init_spaces();
	});

	report_ready();
	return true;
}

这里加一个计时器是为了等待所有的space_server进程注册过来，避免创建场景时找不到可用的space_server进程。极端情况下，这个5s的超时过后space_server进程还没有注册过来，此时需要继续等待：

void space_service::create_init_spaces()
{
	if(m_game_loads.empty())
	{
		add_timer_with_gap(std::chrono::milliseconds(5 * 1000), [=]()
		{
			create_init_spaces();
		});
		return;
	}
	for(const auto& one_pair: m_init_spaces_to_create)
	{
		do_create_space(one_pair.first, one_pair.second, std::string{});
	}
	m_init_spaces_to_create.clear();
}

上面的实现其实也有很大的问题，如果只有一部分的space_server注册过来，这些预创建的场景的压力就全都在少数的space_server进程上了。正常的做法是限定一个space_server的最大负载。如果大于此负载，则后续场景不再创建，继续开启计时器等待，在下次create_init_spaces时继续寻找合适的space_server去消耗m_init_spaces_to_create中剩下的场景。

实际的项目中单space_no的预先创建场景数量并不永远是1，因为单个进程单个场景里的人数承载是有限的，为了处理大量的玩家，一般会给每个space_no来指定需要预先创建的场景的个数，这里的实现只是为了偷懒。

此外预先创建并不只是在服务器启动的时候去执行，还可以在某些需要大量的小场景创建的活动开始之前执行。例如游戏每周五晚上八点会开启某种1v1的匹配活动，每组人员匹配成功之后都会进入一个专属的小场景中进行决斗。由于这个玩法的奖励丰厚，导致参与的玩家非常的多，在八点之后的瞬间就会创建巨量的小场景，此时全服的CPU都会有一个非常明显的上升，出现长时间的卡顿。为了解决这个卡顿问题，我们采取了每周五晚上七点开始慢慢的每隔10s创建一个小场景实例，直到九点钟此玩法结束。这样的慢慢预先创建就达到了削峰的作用，从而解决了瞬间批量创建导致的长时间卡顿。

上面介绍的就是预先创建场景的流程，实际情况下可能会出现玩家人数太多导致场景太拥挤，负载太高的问题。此时我们需要做定期的基于负载均衡的场景数量扩张，如果场景配置数据里开启了auto_create_new_heavy_load的话就会自动执行此流程。这个流程的入口在check_heavy_load_auto_create函数中，这个函数会定期的扫描这些开启了自动扩容的场景里的平均人数负载，如果大于了80%则会自动的创建一个新实例。space_service在初始化的时候会收集这些类型的space，并存储到m_check_load_create_spaces：

bool space_service::init(const json::object_t& data)
{
	// 省略很多代码
	auto temp_row = m_space_config_data->begin_row();
	while(temp_row.valid())
	{
		auto cur_row = temp_row;
		temp_row = m_space_config_data->next_row(temp_row);
		std::uint32_t cur_space_no;
		std::uint32_t cur_space_type;
		if(!cur_row.expect_value(space_no_column, cur_space_no))
		{
			continue;
		}
		if(!cur_row.expect_value(space_type_column, cur_space_type))
		{
			continue;
		}
		auto cur_space_type_info = misc::space_type_info_mgr::get_space_type_info(cur_space_type);
		if(!cur_space_type_info)
		{
			continue;
		}
		if(cur_space_type_info->auto_create_new_heavy_load)
		{
			m_check_load_create_spaces.push_back(std::make_pair(cur_space_no, cur_space_type));
		}
	}
	m_logger->info("auto crate space is {}", serialize::encode(m_check_load_create_spaces).dump())
	add_timer_with_gap(std::chrono::milliseconds(5 * 1000), [=]()
	{
		check_heavy_load_auto_create();
	});
	report_ready();
	return true;
}

void space_service::check_heavy_load_auto_create()
{
	// 会定期的扫描这些开启了自动扩容的场景里的平均人数负载，如果大于了`80%`则会自动的创建一个新实例：
	add_timer_with_gap(std::chrono::milliseconds(5 * 1000), [=]()
	{
		check_heavy_load_auto_create();
	});
	std::vector<std::pair<std::uint32_t, std::uint32_t>> need_create_spaces;
	for(auto [cur_space_no, cur_space_type]: m_check_load_create_spaces)
	{
		auto temp_iter = m_spaces_by_no.find(cur_space_no);
		if(temp_iter == m_spaces_by_no.end())
		{
			continue;
		}
		const auto& cur_space_ids = temp_iter->second;
		int space_instance_count = 0;
		int space_player_count = 0;
		for(const auto& one_space_id: cur_space_ids)
		{
			auto cur_mono_space_instance_iter = m_mono_spaces.find(one_space_id);
			if(cur_mono_space_instance_iter != m_mono_spaces.end())
			{
				space_instance_count++;
				space_player_count += cur_mono_space_instance_iter->second->players.size();
			}
		}
		auto cur_space_type_info = misc::space_type_info_mgr::get_space_type_info(cur_space_type);
		if(cur_space_type_info && space_player_count > (space_instance_count * cur_space_type_info->max_player_load) * 0.8)
		{
			need_create_spaces.push_back(std::make_pair(cur_space_no, cur_space_type));
		}
	}
	m_logger->info("check_heavy_load_auto_create with result {}", serialize::encode(need_create_spaces).dump());
	for(auto one_space_pair: need_create_spaces)
	{
		do_create_space(one_space_pair.first, one_space_pair.second, std::string{}, std::string{}, json::object_t{});
	}
}

场景内实体创建流程

场景创建结束之后这个space_entity就根据自己的space_no对应的配置文件来开启自身独特的逻辑。由于场景作为游戏活动的主要承载容器，根据玩法的不同会执行各种不同的逻辑，因此这些逻辑都放在了space_entity的组件space_component上，space_entity在创建的时候会顺带的初始化这些component:

// bool space_entity::init(const json::object_t& data)
json::object_t components_data;
auto components_data_iter = data.find("components");
if(components_data_iter != data.end())
{
	try
	{
		components_data_iter->second.get_to(components_data);
	}
	catch(const std::exception& e)
	{
		m_logger->error("components data not map");
		return false;
	}
	
	
}
if(!add_components<
	space_cell_component,
	space_navi_component,
	space_event_component,
	space_spawn_component,
	space_match_component,
	space_quest_component
	>(components_data))
{
	m_logger->error("fail to add components ");
	return false;
}

其中最基础的逻辑就是创建场景中的所有entity，因为一个空荡荡的场景实在是毫无可玩性，这部分逻辑由space_spawn_component负责，目前只给space_entity设计了两种可以创建的server_entity，分别是陷阱trap_entity和怪物monster_entity。

在space_spawn_component组件启动的时候会从场景的配置文件中加载这些要创建的陷阱与怪物数据主要是其创建的位置、朝向、类型等信息。由于场景中的每个陷阱和怪物都可能带有独特的逻辑，所以我们创建这些server_entity的时候需要标注其对应的数据表的配置行是哪一个，也就是sid字段，代表这个server_entity的配置表流水号:

struct trap_sysd_columns
{
	typed_matrix::typed_matrix::column_index spawn;
	typed_matrix::typed_matrix::column_index pos;
	typed_matrix::typed_matrix::column_index sid;
	typed_matrix::typed_matrix::column_index trap_type;
	typed_matrix::typed_matrix::column_index trap_radius;
	typed_matrix::typed_matrix::column_index trap_height;
	typed_matrix::typed_matrix::column_index player_trigger;
	typed_matrix::typed_matrix::column_index monster_trigger;
	bool valid() const
	{
		return spawn.valid() && pos.valid() && sid.valid() && trap_type.valid() && trap_radius.valid() && trap_height.valid() && player_trigger.valid() && monster_trigger.valid();
	}
	bool load(const typed_matrix::typed_matrix* trap_sysd);

};

struct monster_sysd_columns
{
	typed_matrix::typed_matrix::column_index spawn;
	typed_matrix::typed_matrix::column_index pos;
	typed_matrix::typed_matrix::column_index sid;
	typed_matrix::typed_matrix::column_index yaw;
	typed_matrix::typed_matrix::column_index no;
	typed_matrix::typed_matrix::column_index name;
	bool valid() const
	{
		return spawn.valid() && pos.valid() && sid.valid() && no.valid() && name.valid() && yaw.valid();
	}
	bool load(const typed_matrix::typed_matrix* trap_sysd);

};
private:
	class rpc_helper;
	trap_sysd_columns m_trap_columns;
	monster_sysd_columns m_monster_columns;
	const typed_matrix::typed_matrix* m_trap_sysd;
	const typed_matrix::typed_matrix* m_monster_sysd;

当space_spawn_component组件被激活的时候开始正式的通过space_traps和space_monster来根据数据来创建server_entity：

bool space_spawn_component::init(const json &data)
{

	m_trap_sysd = m_owner->space_sysd("trap");
	m_monster_sysd = m_owner->space_sysd("monster");

	return m_trap_sysd && m_monster_sysd && m_trap_columns.load(m_trap_sysd) && m_monster_columns.load(m_monster_sysd);
}
void space_spawn_component::activate()
{
	m_owner->logger()->info("space_spawn_component activate");
	if (!spawn_traps())
	{
		m_owner->logger()->error("spawn_traps fail");
	}
	if (!spawn_monsters())
	{
		m_owner->logger()->error("spawn_monsters fail");
	}
}

而根据一行配置数据去创建一个server_entity就是将所有相关配置数据填入初始化参数init_info和创生位置enter_info中，然后调用space_entity上提供的create_entity接口:

json cur_trap_prop;
cur_trap_prop["no"] = temp_trap_type;
cur_trap_prop["sid"] = temp_sid;
cur_trap_prop["trap_height_min"] = temp_trap_height[0];
cur_trap_prop["trap_height_max"] = temp_trap_height[1];
cur_trap_prop["trap_radius"] = temp_trap_radius;
cur_trap_prop["client_visible"] = true;
std::uint64_t cur_entity_flag = 0;
if (temp_player_trigger)
{
	cur_entity_flag |= 1ull << std::uint64_t(enums::entity_flag::is_player);
}
if (temp_monster_trigger)
{
	cur_entity_flag |= 1ull << std::uint64_t(enums::entity_flag::is_monster);
}
cur_trap_prop["trap_cb_any_flag"] = cur_entity_flag;
json::object_t trap_init_info, trap_enter_info;
trap_init_info["prop"] = cur_trap_prop;
trap_init_info["is_ghost"] = false;
trap_init_info["call_proxy"] = "";
trap_enter_info["pos"] = temp_born_pos;
trap_enter_info["yaw"] = 0;
return m_owner->create_entity("trap_entity", m_owner->gen_entity_id(), trap_init_info, trap_enter_info);

这个space_entity::create_entity就是一个对space_server::create_entity的简单封装，主要目的是为了保证每个被创建的actor_entity都能执行到enter_space：

actor_entity* space_entity::create_entity(const std::string& entity_type, const std::string& entity_id, json::object_t& init_info, const json::object_t& enter_info, std::uint64_t online_entity_id)
{

	if(online_entity_id == 0)
	{
		online_entity_id = gen_online_entity_id();
	}

	std::string create_entity_error;
	auto cur_entity = get_server()->create_entity(entity_type, entity_id, online_entity_id, init_info, create_entity_error);
	if(!cur_entity)
	{
		m_logger->error("fail to create_entity type {} id {} with error {}", entity_type, entity_id, create_entity_error);
		return nullptr;
	}
	auto cur_actor_entity = dynamic_cast<actor_entity*>(cur_entity);
	if(!cur_actor_entity)
	{
		m_logger->error("fail to create actor_entity with entity_type {}", cur_entity->type_name());
		get_server()->destroy_entity(cur_entity);
		return nullptr;
	}
	// 省略一些代码
	enter_space(cur_actor_entity, enter_info);
	return cur_actor_entity;
}

场景进出流程

这里的enter_space负责设置actor_entity的位置，并在space_entity上构建每个类型的查找map，方便快速的根据online_entity_id查找当前场景内的某个actor_entity:

void space_entity::enter_space(actor_entity* cur_entity, const json::object_t& enter_info)
{
	
	std::array<double, 3> cur_enter_pos = {0.0};
	double cur_enter_yaw = 0;
	try
	{
		enter_info.at("pos").get_to(cur_enter_pos);
		enter_info.at("yaw").get_to(cur_enter_yaw);
	}
	catch(std::exception& e)
	{
		m_logger->error("enter info doesnt has pos yaw{}", e.what());
	}
	// 省略一些非重点代码
	m_logger->info("entity {} enter space {} pos {}", cur_entity->entity_id(), entity_id(), json(cur_enter_pos).dump());
	cur_entity->set_pos_yaw(cur_enter_pos, cur_enter_yaw);
	m_total_entities[cur_entity->m_base_desc.m_type_id][cur_entity->m_base_desc.m_local_entity_id] = cur_entity;
	if(cur_entity->is_player())
	{
		m_players[cur_entity->m_base_desc.m_local_entity_id] = cur_entity;
	}
	// 省略一些非重点代码
	m_actors_by_online_id[cur_entity->online_entity_id()] = cur_entity;
	m_entity_enter_counter[cur_entity->m_base_desc.m_type_id]++;

	cur_entity->set_space(this);
	auto cur_lambda = [cur_entity](space_component* cur_comp)
	{
		cur_comp->on_enter_space(cur_entity);
	};
	call_component_interface(cur_lambda);
	
}

同时这里先执行actor_entity::set_space来通知这个actor_entity来执行场景绑定，并通知这个actor_entity进入了新的场景, actor_entity所有组件执行on_enter_space。当set_space完成之后，再通知当前场景上的所有space_component来接受新的actor_entity进入：

void actor_entity::set_space(space_entity* in_space)
{
	if(!m_space)
	{
		m_space = in_space;
		enter_space();
	}
	else
	{
		assert(!in_space);
		auto pre_space = m_space;
		m_space = nullptr;
		leave_space(pre_space);
	}
}
class actor_component_interface
{
public:
	virtual void on_leave_space(space_entity* cur_space)
	{

	}
	virtual void on_enter_space()
	{

	}
};

void actor_entity::enter_space()
{
	m_prop_flags = actor_data_prop_queue::get_actor_property_flags();
	auto cur_lambda = [](actor_component* cur_comp)
	{
		cur_comp->on_enter_space();
	};
	call_component_interface(cur_lambda);
}

每个具体的actor_entity都可以复写这个enter_space，例如trap_entity就在enter_space的时候创建了两个AOI区域，来接收其他actor_entity的进出陷阱消息：

void trap_entity::enter_space()
{
	actor_entity::enter_space();
	get_space()->register_sid_entity<trap_entity>(this);
	if(is_ghost())
	{
		return;
	}
	
	auto cur_actor_aoi_comp = get_component<actor_aoi_component>();
	aoi::aoi_radius_controller cur_aoi_ctrl;
	cur_aoi_ctrl.any_flag = m_prop_data.trap_cb_any_flag();
	m_logger->info("trap aoi flag is {}", cur_aoi_ctrl.any_flag);
	cur_aoi_ctrl.need_flag = m_prop_data.trap_cb_need_flag();
	cur_aoi_ctrl.forbid_flag = m_prop_data.trap_cb_forbid_flag();
	cur_aoi_ctrl.radius = m_prop_data.trap_radius();
	cur_aoi_ctrl.min_height = m_prop_data.trap_height_min();
	cur_aoi_ctrl.max_height = m_prop_data.trap_height_max();
	cur_aoi_ctrl.max_interest_in = 30;

	cur_actor_aoi_comp->add_aoi_radius(cur_aoi_ctrl, [this](actor_entity* other, bool is_enter)
	{
		if(is_enter)
		{
			if(m_aoi_in_cb)
			{
				m_aoi_in_cb(other);
			}
		}
		else
		{
			if(m_aoi_leave_cb)
			{
				m_aoi_leave_cb(other->entity_id(), other->aoi_idx(), *other->get_call_proxy());
			}
		}
	}, static_type_name());
}

上面介绍的是由场景负责创建的actor_entity的进入场景流程，由于玩家实体player_entity并不是由场景创建的，因此可以跳过开头的space_entity::create_entity流程，直接切入到space_entity::enter_space来执行，后面会在玩家进出场景中介绍。

当单次陷阱被触发或者怪物被杀死时，他们会主动的调用space_entity上销毁自己的接口：

void space_entity::on_entity_killed(actor_entity* target, actor_entity* killer)
{
	if(!target->is_player())
	{
		// 处理非玩家的死亡逻辑
		get_server()->destroy_entity(target);
		return;
	}
	// 省略非重点代码
}

这里的destroy_entity会调用到server_entity::deactivate，如果当前是actor_entity，则会判断自身是否已经在场景内，如果有的话先执行leave_space操作:

void space_server::destroy_entity(entity::server_entity* cur_entity)
{
	m_logger->info("deactive entity {} with type {}", cur_entity->entity_id(), cur_entity->m_base_desc.m_type_name);
	cur_entity->deactivate();
	m_entities_to_destroy.push_back(cur_entity);
}


void actor_entity::deactivate()
{	
	auto cur_space = get_space();
	if(cur_space)
	{
		cur_space->leave_space(this);
	}
	clear_components();
	m_dispatcher.clear();
	m_misc_dispatcher.clear();
	m_prop_dispatcher.clear();
	m_migrate_in_finish_dispatcher.clear();
	server_entity::deactivate();
}

这里的space_entity::leave_space刚好是之前enter_space的逆操作,会优先通知space_component这个actor_entity的离开，然后再执行actor_entity::set_space(nullptr)来通知到这个actor_entity上的所有actor_component来执行on_leave_space操作，最后再清除当前space_entity上对这个actor_entity的所有记录：

bool space_entity::leave_space(actor_entity* cur_entity)
{
	if(!cur_entity)
	{
		return false;
	}
	
	if(cur_entity->get_space() != this)
	{
		return false;
	}
	auto cur_lambda = [cur_entity](space_component* cur_comp)
	{
		cur_comp->on_leave_space(cur_entity);
	};
	call_component_interface(cur_lambda);
	m_aoi_manager->remove_pos_entity(aoi::aoi_pos_idx{cur_entity->aoi_idx()});
	cur_entity->set_space(nullptr);
	auto cur_entity_id = cur_entity->entity_id();
	// 省略非重点代码
	m_actors_by_online_id.erase(cur_entity->online_entity_id());
	m_entity_leave_counter[cur_entity->m_base_desc.m_type_id]++;
	// 省略非重点代码
	
	return true;
}

场景任务流程

不同的场景提供了不同的游戏体验，玩家、怪物、陷阱等实体都是这些游戏体验中的角色。但是体验中光有角色是远远不够的，好需要有剧本，在剧本中来制定整个体验中各个角色所承担的任务，以及任务之间的关联。由于游戏场景中的策划快速铺量和定制化的需求，这些任务的关联不会直接在代码里实现，而是策划指定一些基础的任务规则需求，然后策划在配置数据中利用这些规则来创建具体的任务流程实例，并增加一些逻辑判定来处理任务之间的关联。这就是基于配表的场景流程，在space_entity中提供了一个space_quest_component组件来对这些任务规则做支持，为了统一管理各项规则，这里给所有的任务规则建立了一个基类space_quest:

class space_quest
{
public:

	typed_matrix::typed_row m_sysd;
	space_quest_component* m_quest_component;
	space_quest(space_quest_component* in_quest_component, typed_matrix::typed_row in_sysd)
	: m_sysd(in_sysd)
	, m_quest_component(in_quest_component)
	{

	}
	virtual bool enter()
	{
		return true;
	}

	virtual void leave()
	{
		
	}
	virtual ~space_quest()
	{

	}
protected:
	void change_to_next(std::uint32_t next_quest_id);

};

这里的space_quest里的m_sysd对应了场景任务表里的一行配置数据，每行配置数据都有一个流水号quest_id来关联，当这个流程被激活的时候会触发其enter()函数来开启自定义逻辑，当流程结束的时候调用leave()函数来做一些清理工作，流程之间的跳转则需要借助change_to_next来执行:

void space_quest::change_to_next(std::uint32_t next_quest_id)
{
	m_quest_component->change_to_quest(next_quest_id);
}

void space_quest_component::change_to_quest(std::uint32_t next_quest_id)
{
	auto pre_quest = m_current_quest;
	m_current_quest = nullptr;
	if(pre_quest)
	{
		pre_quest->leave();
		delete pre_quest;
	}
	auto cur_row = m_quest_sysd->get_row(next_quest_id);
	if(!cur_row.valid())
	{
		return;
	}
	std::string cur_quest_type;
	if(!cur_row.expect_value("quest_type", cur_quest_type))
	{
		m_owner->logger()->error("cant find quest_type for row {}", next_quest_id);
		return;
	}
	if(cur_quest_type == "wait_seconds")
	{
		m_current_quest = new wait_second_quest(this, cur_row);
	}
	else if(cur_quest_type == "kill_all_monsters")
	{
		m_current_quest = new kill_all_monsters_quest(this, cur_row);
	}
	// 省略很多具体的quest子类
	else if(cur_quest_type == "final")
	{
		m_current_quest = new final_quest(this, cur_row);
	}
	if(!m_current_quest)
	{
		m_owner->logger()->error("fail to create quest of type {}", cur_quest_type);
		return;
	}
	m_owner->logger()->info("enter quest type {} id {}", cur_quest_type, next_quest_id);
	
	if(!m_current_quest->enter())
	{
		m_owner->logger()->error("enter quest fail type {} id {}", cur_quest_type, next_quest_id);
	}
	// 先enter 再设置quest 这样等到客户端接收到quest id改变之后 相关的数据已经设置好了
	m_owner->set_quest_id(next_quest_id);
}

在space_quest_component上有m_current_quest字段来记录当前正在执行的任务流程，当要切换任务流程的时候，会先调用之前任务流程的leave，并delete来释放内存。然后再通过读表来获取新流程的类型并创建新的space_quest实例，并调用其enter函数，这样流程的切换就完成了。space_quest_component在激活的时候会默认强制切换到流水号为1的任务上：

bool space_quest_component::init(const json& data)
{
	
	
	m_quest_sysd = m_owner->space_sysd("quest");
	return true;
}

void space_quest_component::activate()
{
	if(m_owner->is_cell_space())
	{
		return;
	}
	change_to_quest(1);

}

但是流程什么时候切换还是得依靠space_quest的具体子类逻辑去执行，space_quest_component无法控制。就以最简单的wait_seconds来说，需要在对应的配置数据里提供好等待时间delay和下一个任务的流水号next：

class wait_second_quest: public space_quest
{
public:
	using space_quest::space_quest;
	bool enter() override
	{
		std::uint32_t next_sid;
		float delay_seconds;
		if(!m_sysd.expect_value("next", next_sid) || !m_sysd.expect_value("delay", delay_seconds))
		{
			return false;
		}
		m_quest_component->get_owner()->add_timer_with_gap(std::chrono::milliseconds(int(delay_seconds*1000)), [this, next_sid]()
		{
			change_to_next(next_sid);
		});
		return true;
	}

};

而对于比较复杂的space_quest，则需要在space_entity上增加各种事件监听函数，来获取当前子任务的进度。就以kill_all_monsters_quest来说，这个任务需要击杀指定流水号集合里的所有怪物，因此自身需要提供一个m_monster_sids来存储还需要击杀的怪物流水号，同时在enter的时候根据配表来初始化这个集合，并在space_entity上注册sid_monster_killed这个事件来对接on_monster_killed函数，从而更新m_monster_sids，并判定当前任务的完成以及执行后续任务的跳转：

class kill_all_monsters_quest: public space_quest
{
	std::unordered_set<std::uint32_t> m_monster_sids;
	std::uint32_t m_next_sid;
	utility::listen_handler<std::string> m_listen_handler;
	private:
	void on_monster_killed(std::uint32_t monster_sid)
	{
		if(m_monster_sids.erase(monster_sid))
		{
			auto cur_space_data_entity = m_quest_component->get_owner()->get_space_data_entity();
			for(const auto& one_pair: cur_space_data_entity->prop_data().quest_monsters())
			{
				if(one_pair.second == monster_sid)
				{
					cur_space_data_entity->prop_proxy().quest_monsters().erase(one_pair.first);
					break;
				}
			}
		}
		if(m_monster_sids.empty())
		{
			change_to_next(m_next_sid);
		}
	}
public:
	using space_quest::space_quest;
	bool enter() override
	{
		if(!m_sysd.expect_value("next", m_next_sid) || !m_sysd.expect_value("monsters", m_monster_sids))
		{
			return false;
		}
		std::function<void(const std::string&, const json&)> cur_lambda = [this](const std::string& event, const json& detail)
		{
			if(detail.is_number_unsigned())
			{
				on_monster_killed(detail.get<std::uint32_t>());
			}
		};
		m_listen_handler = m_quest_component->get_owner()->dispatcher().add_listener(std::string("sid_monster_killed"), cur_lambda);
		return true;
	}

	void leave() override
	{
		m_quest_component->get_owner()->dispatcher().remove_listener(m_listen_handler);
	}
};

当这个kill_all_monsters_quest彻底完成的时候，其leave函数就会删除之前注册的sid_monster_killed事件处理handler，避免其被delete之后还接受新的怪物被击杀事件，引发crash。

在space_quest_component中提供了一些基础的流程定义，后续有需要的话可以非常方便的进行扩展，根据之前的项目经验，大型MMO一般100个流程规则基本就可以覆盖了：

if(cur_quest_type == "wait_seconds")
{
	m_current_quest = new wait_second_quest(this, cur_row);
}
else if(cur_quest_type == "kill_all_monsters")
{
	m_current_quest = new kill_all_monsters_quest(this, cur_row);
}
else if(cur_quest_type == "kill_one_monster")
{
	m_current_quest = new kill_one_monster_quest(this, cur_row);
}
else if(cur_quest_type == "trig_one_trap")
{
	m_current_quest = new wait_one_trap_trig_quest(this, cur_row);
}
else if(cur_quest_type == "trig_all_traps")
{
	m_current_quest = new wait_all_traps_trig_quest(this, cur_row);
}
else if(cur_quest_type == "fill_all_traps")
{
	m_current_quest = new wait_traps_fill_quest(this, cur_row);
}
else if(cur_quest_type == "wait_one_event")
{
	m_current_quest = new wait_one_event_quest(this, cur_row);
}
else if(cur_quest_type == "wait_all_events")
{
	m_current_quest = new wait_all_events_quest(this, cur_row);
}
else if(cur_quest_type == "final")
{
	m_current_quest = new final_quest(this, cur_row);
}

在副本流程逻辑彻底完成之后，m_current_quest应该会切换到final_quest上，这个space_quest什么都不做，不执行任何的跳转，代表终止流程:

class final_quest: public space_quest
{
public:
	using space_quest::space_quest;
};

场景销毁流程

场景流程完成之后，服务器就需要回收这个场景所占据的所有资源。例如在space_entity的leave_space逻辑中，会记录当前剩下的玩家数量，如果剩下的玩家数量变成了0，也会开启一个自动销毁逻辑：

bool space_entity::leave_space(actor_entity* cur_entity)
{
	// 省略一些代码
	if(cur_entity->is_player() && m_total_entities[cur_entity->m_base_desc.m_type_id].empty())
	{
		add_auto_destroy_timer();
	}
	// 省略一些代码
	return true;
}
const std::uint32_t m_auto_desctroy_gap_seconds = 60;

void space_entity::add_auto_destroy_timer()
{
	if(m_auto_destroy_timer.valid())
	{
		return;
	}
	if(!m_space_type_info->is_town_space && !m_space_type_info->is_union_space)
	{
		m_auto_destroy_timer = add_timer_with_gap(std::chrono::milliseconds(m_auto_desctroy_gap_seconds * 1000), [this]()
		{
			auto_destroy();
		});
	}
}

这里设置为了所有玩家离开之后，60秒倒计时销毁：

void space_entity::auto_destroy()
{
	m_auto_destroy_timer.reset();
	if(!m_players.empty())
	{
		// 有玩家 暂时不销毁
		return;
	}
	// 如果场景内没有玩家 则尝试自动销毁

	m_logger->warn("space_entity::auto_destroy");

	utility::rpc_msg cur_request_msg;
	cur_request_msg.cmd = "request_destroy_space";
	cur_request_msg.args.push_back(entity_id());
	cur_request_msg.from = *get_call_proxy();
	get_server()->call_service("space_service", cur_request_msg);

}

这里并不执行真正的销毁逻辑，而是向space_service发起notify_destroy_space这个rpc来通知此场景进程上的space_manager执行场景销毁逻辑。这个notify_destroy_space有一个前提条件，此space内已经没有玩家：

void space_service::request_destroy_space(const utility::rpc_msg &data, const std::string &space_id)
{
	auto cur_space_type_iter = m_space_types.find(space_id);

	utility::rpc_msg cur_msg;

	do
	{
		if (cur_space_type_iter == m_space_types.end())
		{
			cur_msg.err = "invalid space";
			break;
		}
		if (!cur_space_type_iter->second->is_union_space)
		{
			auto cur_space_iter = m_mono_spaces.find(space_id);
			if (cur_space_iter == m_mono_spaces.end())
			{
				cur_msg.err = "invalid space";
				break;
			}
			if (!cur_space_iter->second->players.empty())
			{
				cur_msg.err = "space has player";
				break;
			}
		}
		else
		{
			// 暂时省略另外一个分支
		}
	} while (0);

	cur_msg.cmd = "reply_destroy_space";
	cur_msg.args.push_back(space_id);
	get_server()->call_server(this, data.from, cur_msg);
	if (cur_msg.err.empty())
	{
		destroy_space_impl(space_id, cur_space_type_iter->second);
	}
	else
	{
		m_logger->error("request_destroy_space space_id {} err {}", space_id, cur_msg.err);
	}
}
void space_service::destroy_space_impl(const std::string &space_id, const misc::space_type_info *space_type)
{
	m_logger->info("destroy_space_impl {}", space_id);
	if (!space_type->is_union_space)
	{
		auto cur_space_iter = m_mono_spaces.find(space_id);
		if (cur_space_iter == m_mono_spaces.end())
		{
			return;
		}

		if (!cur_space_iter->second->players.empty())
		{
			return;
		}
		auto cur_game_iter = m_game_loads.find(cur_space_iter->second->game_id);
		if (cur_game_iter == m_game_loads.end())
		{
			return;
		}
		m_spaces_by_no[cur_space_iter->second->space_no].erase(cur_space_iter->first);
		utility::rpc_msg cur_msg;
		cur_msg.cmd = "notify_destroy_space";
		cur_msg.args.push_back(space_id);
		auto cur_game_id = cur_space_iter->second->game_id;
		call_space_manager(cur_game_id, cur_msg);
		m_mono_spaces.erase(cur_space_iter);

		cur_game_iter->second.mono_spaces.erase(space_id);
	}
	else
	{
		// 暂时省略另外一个分支
	}
}

这里的space_manager::notify_destroy_space执行的就很暴力了，通知指定场景内的所有actor执行退出逻辑，然后执行这个space_entity的自毁:

void space_manager::notify_destroy_space(const utility::rpc_msg &data, const std::string &space_id)
{
	auto cur_iter = m_spaces.find(space_id);
	if (cur_iter == m_spaces.end())
	{
		m_logger->error("fail to destroy id {} ", space_id);
		return;
	}
	cur_iter->second->clear_actors();
	m_server->destroy_entity(cur_iter->second);
	m_spaces.erase(cur_iter);
}

这里的clear_actors区分了一下是否是玩家类型，如果不是玩家类型则在退出之后直接销毁自身:

void space_entity::clear_actors()
{
	m_logger->warn("clear_actors");
	for(auto& one_type_ent_vec: m_total_entities)
	{
		std::vector<actor_entity*> temp_actors;
		for(auto one_entity_pair: one_type_ent_vec)
		{
			auto cur_entity = one_entity_pair.second;
			if(!cur_entity)
			{
				continue;
			}
			temp_actors.push_back(cur_entity);
		}
		for(auto one_actor: temp_actors)
		{
			leave_space(one_actor);
			if(!one_actor->is_player() && !one_actor->is_exact_type<space_data_entity>())
			{
				get_server()->destroy_entity(one_actor);
			}
		}
	}
}

对于有些类型的场景来说这个60秒的等待时间可能太长了，特别是一些高频创建和销毁的场景，典型例子就是匹配场景。为了加速场景资源的回收，在比赛结束finish_match中会触发一个十秒的销毁倒计时, 内部会通知space_service来执行request_countdown_destroy：

void space_match_component::finish_match()
{
	m_owner->cancel_timer(m_match_finish_timer);
	m_match_finish_timer.reset();
	m_is_match_finish = true;
	m_owner->get_space_data_entity()->prop_proxy().match_finish().set(true);
	std::uint32_t cur_winner_faction = m_faction_num;
	// 省略一些代码
	utility::rpc_msg finish_msg;
	finish_msg.cmd = "report_match_finish";
	finish_msg.set_args(m_owner->match_uid(), cur_winner_faction, delta_scores);
	m_owner->call_service("match_service", finish_msg);
	utility::rpc_msg countdown_msg;
	countdown_msg.cmd = "request_countdown_destroy";
	std::uint32_t countdown_ts = 10; // 10s之后自动销毁
	countdown_msg.set_args(m_owner->entity_id(), countdown_ts);
	m_owner->call_service("space_service", countdown_msg);
	m_owner->get_space_data_entity()->prop_proxy().destroy_ts().set(utility::timer_manager::now_ts() + 1000 * countdown_ts);

}

这个request_countdown_destroy负责在space_service上开启一个销毁计时器，时间到了之后直接执行之前介绍的destroy_space_impl:

void space_service::request_countdown_destroy(const utility::rpc_msg& data,  const std::string& space_id, std::uint32_t countdown_seconds)
{
	m_logger->info("request_countdown_destroy space_id {} countdown_seconds {}", space_id, countdown_seconds);
	auto cur_space_type_iter = m_space_types.find(space_id);
	if(cur_space_type_iter == m_space_types.end())
	{
		return;
	}
	auto cur_space_type_info = cur_space_type_iter->second;
	if(cur_space_type_info->is_union_space)
	{
		return;
	}
	auto cur_mono_space_iter = m_mono_spaces.find(space_id);
	if(cur_mono_space_iter == m_mono_spaces.end())
	{
		return;
	}
	kick_players_impl(space_id, cur_space_type_info);
	
	add_timer_with_gap(std::chrono::milliseconds(countdown_seconds * 1000), [=]()
	{
		kick_players_impl(space_id, cur_space_type_info);
		destroy_space_impl(space_id, cur_space_type_info);
	});
}

这里在开启计时器之间和计时器到期之后都会执行一下kick_players_impl来通知踢出当前场景里的所有玩家，因为异步过程中可能会出现玩家传送到当前场景的情况，所以这里剔除两次来保证destroy_space_impl执行的时候场景内玩家已经空了。

BigWorld 的场景管理

Cell与Space

在bigworld中每个场景都有一个Space结构来表示，每个Space都有一个uint32的唯一标识符：

class Space
{
public:
	Space( SpaceID id = 0, bool isNewSpace = true,
		bool isFromDB = false, uint32 preferredIP = 0 );
	~Space();

	void shutDown();

	SpaceID id() const		{ return id_; }

	CellData * addCell( CellApp & cellApp, CellData * pCellToSplit = NULL );
	CellData * addCell();
	void addCell( CellData * pCell );
	CellData * addCellTo( CellData * pCellToSplit );
private:
	SpaceID	id_;
	Cells cells_;

	CM::BSPNode * pRoot_;
};

然后对于分布式的场景，整个逻辑场景会由多个方块场景聚合而成，每个方块部分对应一个CellData，然后所有的方块存储在Cells这个CellData的线性容器中：

class Cells
{
private:
	typedef BW::vector< CellData * > Container;

public:
	Cells() {}
	~Cells();

	void add( CellData * pData )		{ cells_.push_back( pData ); }
	void erase( CellData * pData );
private:
	Container cells_;
};

值得注意的是每个CellData除了在这个Cells里线性存储之外，CellData其实还有一个二叉树状结构，它继承自BSPNode，这个BSP其实就是Binary Space Partitioning的简称。每个Space都有一个CM::BSPNode * pRoot_的成员变量来存储二叉分割树的根节点，同时每个BSPNode都有一个BW::Rect range_代表当前Cell负责的场景区域：

class CellData : public CM::BSPNode
{
public:
	CellData( CellApp & cellApp, Space & space );
	CellData( Space & space, BinaryIStream & data );
	~CellData();
};

class BSPNode : public WatcherProvider
{
public:
	BSPNode( const BW::Rect & range );
	virtual ~BSPNode() {};
protected:
	BW::Rect range_;
	EntityBoundLevels entityBoundLevels_;
	BW::Rect chunkBounds_;
};

在二维平面里的二叉分割允许使用任意的直线，不过这里的Binary Space Partitioning会限制为只能水平划分或者垂直划分，对应的addCell接口里需要显示用bool isHorizontal来表明是水平划分还是垂直划分:

virtual CM::BSPNode * addCell( CellData * pCell, bool isHorizontal );

此时就退化成了一个KDTree:

下面就是一个具体按照水平或者竖直方向进行划分的的Space实例:

此时对应的KDTree就是这样的：

注意到前面addCell的时候，新的Cell对应的分割轴上的区间大小其实是0，也就是说新Cell对应的Rect面积是0。新添加的Cell的Rect会在后续的负载均衡中进行调整：

CM::BSPNode * CellData::addCell( CellData * pCell, bool isHorizontal )
{
	const float partitionPt = range_.range1D( isHorizontal ).max_;
	BW::Rect newRange = range_;
	newRange.range1D( isHorizontal ).min_ = partitionPt;
	newRange.range1D( isHorizontal ).max_ = partitionPt;
	pCell->setRange( newRange );

	// TODO: At the moment, the new cell is always added to the right or top. It
	// may be better to choose the side based on which side is unbounded. A
	// simple test might be to check if fabs( min_ ) < fabs( max_ ) of
	// range_.range1D( isHorizontal ).

	return new CM::InternalNode( this, pCell,
			isHorizontal, range_, partitionPt );
}

注意这里最后的返回值是CM::InternalNode，这个类型也继承自BSPNode，传入的两个CellData会作为当前InternalNode的左右子节点存在：

InternalNode::InternalNode( BSPNode * pLeft, BSPNode * pRight,
		bool isHorizontal, const BW::Rect & range, float position ) :
	// Note: There are three constructors.
	BSPNode( range )
{
	this->init();
	pLeft_ = pLeft;
	pRight_ = pRight;
	isHorizontal_ = isHorizontal;
	position_ = position;
}

所以整个BSPNode被划分为了两种类型：

1. 一种是有两个子节点的InternalNode类型，是BSP树里的内部节点，这个类型不负责具体的场景区域，
2. 另外一种是没有子节点的CellData类型，是BSP树里的叶子节点，每个叶子节点负责一块具体的场景区域

Space里存储的根节点CM::BSPNode * pRoot_则可能是两种节点类型中的一种。

Space的创建

在Bigworld里，CellAppMgr负责创建Space，并将其分配到合适的CellApp上运行。这个创建Space的入口函数是createEntityInNewSpace，这个函数会在CellAppMgr收到创建新Space的请求时被调用：

void CellAppMgr::createEntityInNewSpace( const Mercury::Address& srcAddr,
		const Mercury::UnpackedMessageHeader& header,
		BinaryIStream & data )
{
	bool doesSpaceHavePreferredIP;

	data >> doesSpaceHavePreferredIP;

	uint32 preferredIP = (doesSpaceHavePreferredIP ? srcAddr.ip : 0);

	if (doesSpaceHavePreferredIP)
	{
		TRACE_MSG( "CellAppMgr::createEntityInNewSpace: "
					"Creating space with preferred IP %s\n",
				srcAddr.ipAsString() );
	}

	Space * pSpace = new Space( this->generateSpaceID(),
		/*isNewSpace*/ true, /*isFromDB*/ false,
		preferredIP );
	if (pSpace->addCell())
	{
		this->addSpace( pSpace );
	}
	else
	{
		ERROR_MSG( "CellAppMgr::createEntityInNewSpace: "
				"Unable to add a cell to space %u.\n", pSpace->id() );
		bw_safe_delete( pSpace );
	}

	//passing pSpace==NULL is needed here to send the errors (and is safe)
	this->createEntityCommon( pSpace, srcAddr, header, data );
}

这个RPC的第一个参数是bool doesSpaceHavePreferredIP，表示是否要求在RPC发送者的IP地址上创建Space。如果为false，则会将这个preferredIP设置为0，代表随机选择一个CellApp作为Space的CellApp。然后CellAppMgr会通过generateSpaceID来生成一个随机生成的SpaceID作为唯一标识符，并以这些参数来New一个新的Space对象:

Space::Space( SpaceID id, bool isNewSpace, bool isFromDB, uint32 preferredIP ) :
	id_( id ),
	pRoot_( NULL ),
	isBalancing_( false ),
	preferredIP_( preferredIP ),
	isFirstCell_( isNewSpace ),
	isFromDB_( isFromDB ),
	hasHadEntities_( !isFromDB ),
	waitForChunkBoundUpdateCount_( 0 ),
	spaceGrid_( 0.f ),
	spaceBounds_( 0.f, 0.f, 0.f, 0.f ),
	artificialMinLoad_( 0.f )
{
}

在Space的构造函数里，pRoot_被初始化为nullptr, 为了维持树结构的有效性，Space被CellAppMgr创建的时候会自动的通过addCell接口来创建根节点：

CellData * Space::addCell()
{
	CellAppGroup * pGroup = NULL;

	if (!cells_.empty())
	{
		pGroup = cells_.front()->cellApp().pGroup();
	}

	const CellApps & cellApps = CellAppMgr::instance().cellApps();
	CellApp * pCellApp = cellApps.findBestCellApp( this, pGroup );


	return pCellApp != NULL ? this->addCell( *pCellApp ) : NULL;
}

Space::addCell这个接口会通过findBestCellApp选择一个负载合适的CellApp来承载这个完整的Space。然后再以这个pCellApp作为唯一参数去调用双参数形式的addCell，此时第二个参数默认为nullptr:

CellData * Space::addCell( CellApp & cellApp, CellData * pCellToSplit = NULL )
{
	INFO_MSG( "Space::addCell: Space %u. CellApp %u (%s)\n",
			id_, cellApp.id(), cellApp.addr().c_str() );

	if (cellApp.isRetiring())
	{
		WARNING_MSG( "Space::addCell: Adding a cell to CellApp %u (%s) which "
			"is retiring.\n", cellApp.id(), cellApp.addr().c_str() );
	}

	CellData * pCellData = new CellData( cellApp, *this );

	if (pCellToSplit)
	{
		MF_ASSERT( pRoot_ != NULL );
		pRoot_ = pRoot_->addCellTo( pCellData, pCellToSplit );
		MF_ASSERT( pRoot_ != NULL );
	}
	else
	{
		pRoot_ = (pRoot_ ? pRoot_->addCell( pCellData ) : pCellData);
	}

	pRoot_->updateLoad();
	// 省略后续代码
}

这里的会发现此时的pRoot_为空，因此直接使用新创建的pCellData作为pRoot_，因此刚创建的时候Space的BSP树只有一个叶子节点CellData，负责所有区域。

在初始状态下，Space的BSP树只有一个叶子节点CellData，负责所有区域，后续会根据负载均衡的结果来不断的调整BSP树的结构，来增减CellData节点。这部分的内容将留到后续的章节中介绍。

目前执行这个远程调用的代码只有一处，在BaseApp暴露给Python脚本的Base::py_createInNewSpace里:

/**
 *	This method implements the base's script method to create an associated
 *	entity on a cell in a new space.
 */
PyObject * Base::py_createInNewSpace( PyObject * args, PyObject * kwargs )
{
	const char * errorPrefix = "Base.createEntityInNewSpace: ";

	PyObject * pPreferThisMachine = NULL;

	static char * keywords[] = 
	{
		const_cast< char * >( "shouldPreferThisMachine" ),
		NULL
	};

	if (!PyArg_ParseTupleAndKeywords( args, kwargs,
		"|O:Base.createEntityInNewSpace", keywords, &pPreferThisMachine ))
	{
		return NULL;
	}

	std::auto_ptr< Mercury::ReplyMessageHandler > pHandler(
		this->prepareForCellCreate( errorPrefix ) );

	if (!pHandler.get())
	{
		return NULL;
	}

	bool shouldPreferThisMachine = false;

	if (pPreferThisMachine)
	{
		shouldPreferThisMachine = PyObject_IsTrue( pPreferThisMachine );
	}

	Mercury::Channel & channel = 
		BaseApp::getChannel( BaseApp::instance().cellAppMgrAddr() );

	// We don't use the channel's own bundle here because the streaming might
	// fail and the message might need to be aborted halfway through.
	std::auto_ptr< Mercury::Bundle > pBundle( channel.newBundle() );

	// Start a request to the Cell App Manager.
	pBundle->startRequest( CellAppMgrInterface::createEntityInNewSpace,
			pHandler.get() );

	*pBundle << shouldPreferThisMachine;

	*pBundle << this->channel().version();

	*pBundle << false; /* isRestore */

	// See if we can add the necessary data to the bundle
	if (!this->addCellCreationData( *pBundle, errorPrefix ))
	{
		isCreateCellPending_ = false;
		isGetCellPending_ = false;

		return NULL;
	}

	// Send it to the Cell App Manager.
	channel.send( pBundle.get() );
	pHandler.release(); // Now owned by Mercury.

	Py_RETURN_NONE;
}

这个接口会暴露给Python脚本调用，从而创建一个新的Space，并在这个Space里创建一个新的实体。这个接口唯一的参数是shouldPreferThisMachine，表示是否要求在当前BaseApp所在的机器上创建Space。如果为true，则会将当前机器的IP地址传递给CellAppMgr。在选择合适的CellApp的时候会通过BaseCellTrafficScorer来提升指定IP的CellApp的优先级:

/**
 *	This method calculates the score for a CellApp's base-to-cell traffic.
 *	This is determined by comparing the IP address of the CellApp with the
 *	preferred IP of the space on which a new cell is being added. If this
 *	CellApp is running on the preferred machine, then it is likely that many
 *	of the space's Base entities will exist on that machine. This means that
 *	much of the base-to-cell traffic will occur between processes on the same
 *	machine, reducing network load.
 *	This method returns 1 if the CellApp is on the preferred IP, and 0 if not.
 */
float BaseCellTrafficScorer::getScore( const CellApp * pApp,
		const Space * pSpace ) const
{
	MF_ASSERT( pSpace );

	return (pApp->addr().ip == pSpace->preferredIP()) ? 1.f : 0.f;
}

这个时候大家可能有点疑问了，BaseApp上只能管理Base，是不能管理Cell的，那为什么要通知CellAppMgr优先使用当前BaseApp的IP呢？其实BaseApp与CellApp只是进程之间隔离，并不需要使用机器来隔离，一个物理机器上可以同时部署多个BaseApp和CellApp。所以BaseApp暴露自己的IP给CellAppMgr去创建Space是没有什么问题的，这样做的好处就是CellApp与相关的BaseApp之间通信延迟会大大减小，因为只需要本机通信即可。

Space的销毁

Space的销毁同样是由CellAppMgr来负责的，CellAppMgr会收到一个远程调用shutDownSpace，这个调用会传入需要销毁的SpaceID，然后通过findSpace找到对应的Space对象，然后调用其shutDown接口来销毁：

/**
 *	This method handles a message informing us to shut down a space.
 */
void CellAppMgr::shutDownSpace(
		const CellAppMgrInterface::shutDownSpaceArgs & args )
{
	Space * pSpace = this->findSpace( args.spaceID );

	if (pSpace)
	{
		if (pSpace->hasHadEntities())
		{
			// Delay shutting down the space until the end of tick
			//	don't shutdown twice
			if (spacesShuttingDown_.insert( args.spaceID ).second)
			{
				pSpace->shutDown();
			}
		}
		else
		{
			NOTICE_MSG( "CellAppMgr::shutDownSpace: Not shutting down space "
								"%u since it has not had any entities\n",
							pSpace->id() );
		}
	}
	else
	{
		ERROR_MSG( "CellAppMgr::shutDownSpace: Could not find space %u\n",
			args.spaceID );
	}
}

这里的spacesShuttingDown_是一个std::set< SpaceID >，用来记录正在销毁的Space，防止重复销毁。

在执行Space::shutDown的时候，会遍历所有的Cell，并通知其CellApp来销毁Space:

/**
 *	This method shuts down this space and removes it from the system.
 */
void Space::shutDown()
{
	INFO_MSG( "Space::shutDown: Shutting down space %u "
				"(remaining cells: %" PRIzu ")\n",
			id_, cells_.size() );

	Cells::iterator iter = cells_.begin();

	while (iter != cells_.end())
	{
		CellApp * pApp = (*iter)->pCellApp();

		if (pApp)
		{
			pApp->shutDownSpace( this->id() );
		}

		++iter;
	}
}

这里的CellApp::shutDownSpace接口会将销毁Space的请求构造为CellAppInterface::shutDownSpace消息，然后发送给对应的CellApp:

/**
 *	This method lets the CellApp know that the space is being destroyed.
 */
void CellApp::shutDownSpace( SpaceID spaceID )
{
	Mercury::Bundle & bundle = this->bundle();
	bundle.startMessage( CellAppInterface::shutDownSpace );
	bundle << spaceID;

	this->send();
}

当CellApp收到CellAppInterface::shutDownSpace消息的时候，会调用Space::shutDownSpace接口来销毁Space。这里并不会执行立即销毁，而是注册一个定时器shuttingDownTimerHandle_，计时器的超时时间为1s：

/**
 *	This method handles a message from the CellAppMgr telling us that the space
 *	has been destroyed. It may take some time before all the cells are removed.
 */
void Space::shutDownSpace( BinaryIStream & data )
{
	if (!shuttingDownTimerHandle_.isSet())
	{
		// Register a timer to go off in one second.
		shuttingDownTimerHandle_ =
			CellApp::instance().mainDispatcher().addTimer( 1000000, this, NULL,
			"ShutdownSpace" );
	}
	else
	{
		INFO_MSG( "Space::shutDownSpace: Already shutting down.\n" );
	}
}

这个销毁计时器超时之后，会调用pCell_->onSpaceGone接口来通知Cell开始执行退出逻辑，然后检查是否还有其他Cell在Space中，如果没有其他Cell在Space中且Space中没有其他实体在存在，那么就会调用CellApp::destroyCell来彻底销毁Cell：

/**
 *	This method handles the timer associated with the space.
 *	Currently it is only used for the shutting down timer.
 */
void Space::handleTimeout( TimerHandle handle, void * arg )
{
	if (pCell_)
	{
		pCell_->onSpaceGone();

		if (this->hasSingleCell() && entities_.empty())
		{
			CellApp::instance().destroyCell( pCell_ );
			// when the cell is destructed it will clear our ptr to it
			MF_ASSERT( pCell_ == NULL );
		}
	}
}

这里的onSpaceGone接口会遍历所有的实体，调用实体的onSpaceGone脚本接口，然后检查实体是否需要被销毁。如果实体需要被销毁且是RealEntity，那么就会调用实体的destroy接口来销毁实体:

/**
 *	This method is called when this space wants to be destroyed.
 */
void Cell::onSpaceGone()
{
	BW::vector< EntityPtr > entities( realEntities_.size() );
	std::copy( realEntities_.begin(), realEntities_.end(), entities.begin() );

	BW::vector< EntityPtr >::iterator iter = entities.begin();

	while (iter != entities.end())
	{
		EntityPtr pEntity = *iter;

		if (!pEntity->isDestroyed())
		{
			Entity::nominateRealEntity( *pEntity );

			PyObject * pMethod =
				PyObject_GetAttrString( pEntity.get(), "onSpaceGone" );
			Script::call( pMethod, PyTuple_New( 0 ),
					"onSpaceGone", true/*okIfFnNull*/ );

			if (!pEntity->isDestroyed() &&
					pEntity->isReal() &&
					&pEntity->space() == &this->space())
			{
				pEntity->destroy();
			}

			Entity::nominateRealEntityPop();
		}

		++iter;
	}
}

/**
 *	This method kills a cell.
 */
void CellApp::destroyCell( Cell * pCell )
{
	cells_.destroy( pCell );
}

void Cells::destroy( Cell * pCell )
{
	Container::iterator iter = container_.find( pCell->spaceID() );

	MF_ASSERT( iter != container_.end() );

	if (iter != container_.end())
	{
		container_.erase( iter );
		delete pCell;
	}
	else
	{
		ERROR_MSG( "Cells::deleteCell: Unable to kill cell %u\n",
									pCell->spaceID() );
	}

}


/**
 *	The destructor for Cell.
 */
Cell::~Cell()
{
	TRACE_MSG( "Cell::~Cell: for space %u\n", space_.id() );

	while (!realEntities_.empty())
	{
		int prevSize = realEntities_.size();

		realEntities_.front()->destroy();

		MF_ASSERT( prevSize > (int)realEntities_.size() );

		if (prevSize <= (int)realEntities_.size())
		{
			break;
		}
	}

	bw_safe_delete( pReplayData_ );

	MF_ASSERT_DEV( space_.pCell() == this );

	space_.pCell( NULL );
}

在CellAppMgr帧末尾的时候，会遍历spacesShuttingDown_集合，来强行删除所有的成员Space：

{
	SpaceIDs::iterator iter = spacesShuttingDown_.begin();
	while (iter != spacesShuttingDown_.end())
	{
		Spaces::iterator found = spaces_.find( *iter );
		if (found != spaces_.end())
		{
			delete found->second;
			spaces_.erase( found );
		}
		++iter;
	}
	spacesShuttingDown_.clear();
}

看上去一旦接收到销毁Space的请求，就会立即添加到spacesShuttingDown_集合中，然后在CellAppMgr帧末尾的时候，会遍历spacesShuttingDown_集合，来强行删除所有的成员Space。完全没有等待所有的CellSpace销毁完成的步骤，由于CellSpace的销毁是异步的，所以在CellAppMgr帧末尾的时候，可能还有CellSpace在CellApp里处于销毁中的状态。

为了避免异步操作可能出现的问题，需要在销毁的RPC发起者那里确保后续不再需要这些CellSpace去执行逻辑。目前这个shutDownSpace的唯一调用位置就在CellApp上的Space::requestShutDown接口中：

/**
 *
 */
void CellAppMgrGateway::shutDownSpace( SpaceID spaceID )
{
	CellAppMgrInterface::shutDownSpaceArgs args;
	args.spaceID = spaceID;

	channel_.bundle() << args;

	channel_.send();
}


/**
 *	This method sends a request to the CellAppMgr to shut this space down.
 */
void Space::requestShutDown()
{
	if ( CellAppConfig::useDefaultSpace() && this->id() == 1 )
	{
		ERROR_MSG( "Space::requestShutDown: Requesting shut down for "
			"the default space\n" );
	}
	CellApp::instance().cellAppMgr().shutDownSpace( this->id() );
}

这个接口有两个调用位置，一个是Entity::destroySpace，这个destroySpace暴露给了Python，用来让逻辑层来强行驱动一个场景的销毁；另一个是Space::checkForShutDown，用来检查场景是否需要被销毁。

/*~ function Entity destroySpace
	*  @components{ cell }
	*	This method attempts to shut down the space that the entity is in.
	*	It is not possible to shut down the default space.
	*/
PY_METHOD( destroySpace )

/**
 *	This method allows script to destroy a space.
 *
 *	@return Whether we were allowed to destroy the space
 */
bool Entity::destroySpace()
{
	AUTO_SCOPED_THIS_ENTITY_PROFILE;

	if ( CellAppConfig::useDefaultSpace() && this->space().id() == 1)
	{
		PyErr_Format( PyExc_ValueError,
			"destroySpace called on entity %d in default space", int(id_) );
		return false;
	}
	this->space().requestShutDown();
	return true;
}

/**
 *	This method checks whether we should request for this space to shut down.
 *	If we have no entities and we're the only cell, request a shutdown.
 *	We won't actually be deleted however until we've unloaded all our chunks.
 */
void Space::checkForShutDown()
{
	if (this->hasSingleCell() &&
			entities_.empty() && CellApp::instance().hasStarted() &&
			!this->isShuttingDown() &&
			!CellApp::instance().mainDispatcher().processingBroken() &&
			!(CellAppConfig::useDefaultSpace() && id_ == 1)) // Not for the default space.
	{
		INFO_MSG( "Space::checkForShutDown: Space %u is now empty.\n", id_ );
		this->requestShutDown();
	}
}

这个Space::checkForShutDown的销毁条件比较符合预期，就是当前场景没有实体，也只有一个Cell在运行，且不是默认场景。所以此时通知CellAppMgr来销毁这个场景是没有任何问题的。所以这个checkForShutDown接口的调用时机就是两个地方：一个是Space::removeEntity，代表一个Entity离开场景的时候；以及Space::updateGeometry，代表Cell格局被修改的时候:

/**
 *	This method removes an entity from this space.
 */
void Space::removeEntity( Entity * pEntity )
{
	// 省略一些代码

	if (entities_.empty())
	{
		if (pCell_ != NULL)
		{
			this->checkForShutDown();
		}
	}
}

/**
 *	This method handles a message from the server that updates geometry
 *	information.
 */
void Space::updateGeometry( BinaryIStream & data )
{
	bool wasMulticell = !this->hasSingleCell();
	// 省略一些代码
	
	// see if we want to expressly shut down this space now
	if (wasMulticell)
	{
		this->checkForShutDown();
	}
}

网络通信

本章介绍一下如何使用asio进行网络通信， 介绍tcp与udp的区别，并手动实现一个加密连接的http 代理服务器，最后比较一下tcp与udp的差异.

IO模型

本项目的网络通信功能建立在cpp语言里使用最广的网络库Asio之上，此网络库同时支持了阻塞网络模型与异步网络模型：

1. 阻塞 blocking ,当前线程发出IO请求后阻塞在等待IO就绪，然后再发去数据复制请求，然后再阻塞在等待数据拷贝完成；
2. 异步 asynchronouse，线程提交IO请求之后直接返回，系统在执行完IO请求并复制到用户提供的数据区之后再通知完成 这两种网络模型在代码编写时有很大的差别，这里我们用一个非常简单的Echo网络程序来说明一下他们之间的差异：

同步阻塞通信模型

首先我们用asio构造一个同步通信的Echo客户端，这个客户端的负责读取一行用户提供的输入，传输到服务器上，然后等待服务器发回这段输入，接收完数据之后就退出。

const int max_length = 1024;
int main(int argc, char* argv[])
{
	try
	{
		if (argc != 3)
		{
		// 命令行需要传递两个参数 第一个为服务器的ip 第二个为服务器的端口
			std::cerr << "Usage: blocking_tcp_echo_client <host> <port>\n";
			return 1;
		}

	// 构造一个asio的执行环境
		asio::io_context io_context;
	// 使用执行环境构造一个tcp的socket
		tcp::socket s(io_context);
	// resolver负责解析服务器 作用是将 类似于 www.baidu.com:80这样的网址 解析到对应的ip地址和tcp端口
		tcp::resolver resolver(io_context);
	// 将构造的socket连接到指定的服务器
		asio::connect(s, resolver.resolve(argv[1], argv[2]));

		std::cout << "Enter message: ";
		char request[max_length];
		std::cin.getline(request, max_length);
		size_t request_length = std::strlen(request);
	// 将输入的string 构造一个buffer 然后通过之前构造的socket 将这个buffer里的所有数据发送到连接到的服务器
	// 发送期间 当前程序阻塞 直到发送完成或者报错
		asio::write(s, asio::buffer(request, request_length));

		char reply[max_length];
	// 构造一个读取数据的buffer 然后等待服务器发送数据过来
		size_t reply_length = asio::read(s,
				asio::buffer(reply, request_length));
		std::cout << "Reply is: ";
	// 输出服务器发送过来的信息
		std::cout.write(reply, reply_length);
		std::cout << "\n";
	}
	catch (std::exception& e)
	{
	// asio 提供的resolve connect read write等操作都是通过系统提供的相关接口执行的 
	// 如果这些接口返回了错误， asio则会将这些错误转换为异常 抛出
		std::cerr << "Exception: " << e.what() << "\n";
	}

	return 0;
}

对应的Echo同步服务器则需要在特定Tcp端口上开启监听，等待客户端进行连接，读取数据之后再往客户端发回去：

const int max_length = 1024;

void session(tcp::socket sock)
{
	
	try
	{
	// 当一个客户端连接过来的时候 开启这个无限循环
		for (;;)
		{
			char data[max_length];

			std::error_code error;
		// 这里调用read_some来读取客户端连接发送过来的数据到data构造的buffer， 
		// 如果调用出错 则将错误码写入error参数
		// 调用成功则返回读取的字节数量
			size_t length = sock.read_some(asio::buffer(data), error);
			if (error == asio::error::eof)
				break; // Connection closed cleanly by peer.
			else if (error)
				throw std::system_error(error); // Some other error.
		// 将读取的数据再写回客户端
			asio::write(sock, asio::buffer(data, length));
		}
	}
	catch (std::exception& e)
	{
	// 为了避免一个客户端连接出异常导致服务器崩溃 这里使用try将异常打印出来 然后结束循环
		std::cerr << "Exception in thread: " << e.what() << "\n";
	}
}

void server(asio::io_context& io_context, unsigned short port)
{
	// 这里的tcp:v4()返回的就是本机所有IPV4地址 等价于0.0.0.0
	// 这里构造一个tcp::acceptor的监听结构 开启对localhost:port的端口监听
	tcp::acceptor a(io_context, tcp::endpoint(tcp::v4(), port));
	for (;;)
	{
	// 这里的a.accept是一个阻塞调用 当客户端连接到此服务器时
	// 返回此连接对应的socket 
	// accept返回后会构造一个thread 来执行session(socket)这个函数
	// thread构造好之后 开启另外一个线程进行执行 同时detach 避免阻塞当前线程
		std::thread(session, a.accept()).detach();
	// 有多少个同时活动的客户端连接 就会有多少个额外线程
	}
}

int main(int argc, char* argv[])
{
	try
	{
		if (argc != 2)
		{
		// 
			std::cerr << "Usage: blocking_tcp_echo_server <port>\n";
			return 1;
		}
	// asio的网络功能依赖于io_context作为执行环境
		asio::io_context io_context;

		server(io_context, std::atoi(argv[1]));
	}
	catch (std::exception& e)
	{
		std::cerr << "Exception: " << e.what() << "\n";
	}

	return 0;
}

从上述的Echo代码样例可以看出，同步网络编程是从逻辑结构上来说非常简单的，这里的read, write基本可以等价于在执行cin, cout, 函数返回时即可认为对应数据操作已经完成。但是这种逻辑上的简单也有其附加的代价，这几个接口的调用期间，所在线程是完全阻塞住的，无法执行其他任务。客户端程序能够接受这种阻塞，但是对于服务器来说，这样的阻塞是不可接受的，因为服务器要同时服务多个客户端。所以这里服务器每次接收到一个客户端连接时，都会创建一个额外的线程来处理这个客户端连接的所有逻辑。创建线程是一个消耗很大的函数，线程太多也会让系统的线程调度器负担增大。因此生产环境面向并发的网络程序基本不会采用阻塞的网络通信模型。

异步通信模型

由于异步通信模型会导致代码量增加很多，因此这里只提供异步的Echo服务器端的代码展示，上一节中的同步阻塞客户端仍然可以连接到新的异步服务器。在使用异步的监听服务器时，我们使用一个session的结构来管理一个客户端连接, 而不是上一节中给每个客户端连接分配一个线程。

	class session
	: public std::enable_shared_from_this<session>
{
public:
	session(tcp::socket socket)
		: socket_(std::move(socket))
	{
	}

	void start()
	{
		do_read();
	}

private:
	void do_read()
	{
		auto self(shared_from_this());
	// 这里提供一个buffer 然后在对应的socket上发起一个异步读取的操作
	// async_read_some这个操作会立即返回 等到socket接收到一些数据或报错的时候 
	// 才执行我们提供的lambda函数
		socket_.async_read_some(asio::buffer(data_, max_length),
				[this, self](std::error_code ec, std::size_t length)
				{
					if (!ec)
					{
						do_write(length);
					}
				});
	}

	void do_write(std::size_t length)
	{
		auto self(shared_from_this());
	// 这里提供一个buffer 然后在对应的socket上发起一个异步发送的操作
	// async_write这个操作会立即返回 等到socket发送完buffer指定的数据或报错的时候 
	// 才执行我们提供的lambda函数
		socket_.async_write(asio::buffer(data_, length),
				[this, self](std::error_code ec, std::size_t /*length*/)
				{
					if (!ec)
					{
			// 如果没有报错 则继续执行do_read
						do_read();
					}
				});
	}

	tcp::socket socket_;
	enum { max_length = 1024 };
	char data_[max_length];
};

相对于之前同步阻塞的数据收发接口，新的异步收发接口都有一个async_的前缀，同时函数都有一个额外参数来接受一个能转换为std::function<void(std::error_code, std::size_t)>类型的回调函数。这些异步收发接口调用之后会立即返回，不会去等待对应的操作执行结束，只是往asio::io_context发起这个操作并注册这个操作执行结束（包括失败）时的回调函数。在开启多线程处理同一个asio:io_context的情况下，这个回调函数的不保证在发起对应操作的线程上执行，因此逻辑层需要自己处理好回调函数的多线程数据读写问题。值得注意的是这个session结构继承自std::enable_shared_from_this<T>，这个enable_shared_from_this父类的存在导致session的实例必须通过std::make_shared的形式进行创建。同时这个父类还提供了shared_from_this()接口来获取当前this指针对应的shared_ptr<session>。每次发起一个异步操作时我们都通过auto self(shared_from_this())来构造当前实例的一个shared_ptr，然后传递到异步函数的回调lambda中。这样就可以保证异步函数回调时session的生命周期仍然是有效的，对应的由data_构造的buffer也是有效的。

有了这个session结构去管理客户端连接之后，服务器的监听逻辑如下：

class server
{
public:
	server(asio::io_context& io_context, short port)
		: acceptor_(io_context, tcp::endpoint(tcp::v4(), port))
	{
		do_accept();
	}

private:
	void do_accept()
	{
		acceptor_.async_accept(
				[this](std::error_code ec, tcp::socket socket)
				{
					if (!ec)
					{
						std::make_shared<session>(std::move(socket))->start();
					}

					do_accept();
				});
	}

	tcp::acceptor acceptor_;
};

int main(int argc, char* argv[])
{
	try
	{
		if (argc != 2)
		{
			std::cerr << "Usage: async_tcp_echo_server <port>\n";
			return 1;
		}

		asio::io_context io_context;

		server s(io_context, std::atoi(argv[1]));

		io_context.run();
	}
	catch (std::exception& e)
	{
		std::cerr << "Exception: " << e.what() << "\n";
	}

	return 0;
}

这里使用一个server结构体去管理监听，核心逻辑就在do_accept里，内部发起一个async_accept的异步监听操作，每次一个新的客户端连接上来的时候都会执行这个监听函数的回调函数。回调函数负责使用make_shared将acceptor创建的新socket构造出一个新的session，并立即启动这个session。此时server并没有保存session的shared_ptr，这个session的生命周期完全由session内部逻辑去管理，所以session内每个异步操作的回调lambda都需要去捕获这个session的一个shared_ptr以维持引用计数。

TCP的封包与解包

上面的Echo程序是一个使用Asio编写的非常简单的TCP网络通信例子，在这个例子中服务器接收到任意字节数量的数据之后并不做任何逻辑处理直接往回转发。但是在真正有意义的通信程序中，这种无视传输内容的通信是不存在的。在业务层看来，客户端与服务器之间通信传输的是一个个业务层的数据包，由于业务逻辑的不同会导致不同的数据包的大小各不一样。而TCP协议是基于数据流的协议，它的write接口发送的是一段字节，他的read接口获取也是一段字节，这些字节片段基本不可能业务层传递过来的数据包Packet一一对应。

每次read接口读取过来的一段数据就是多个包按序组成的字节流的一部分，这段数据可能不足以包含一个包，也可能包含多个包。所以在TCP接收到数据到将Packet传递给业务逻辑处理中间，我们需要执行一个解包的过程。解包首先要明确的是每一个Packet的边界，即获取下一个完整数据包的大小。为了达到此目的，我们一般在接收到业务层发送一个Packet字节片段的请求时，构造一个新的Packet，这个新Packet的前四个字节用原始Packet的大小进行填充，然后再拼接原始Packet的所有字节到这四个字节之后。这样处理之后，TCP网络解包的流程就大概等价于下面的过程：

#define MAX_PACKET_SZ  65536
char buffer[MAX_PACKET_SZ];
std::function<void(const char* , std::uint32_t)> packet_callback;
std::uint32_t buffer_begin = 0;
asio::tcp::socket tcp_socket;
while(true)
{
	//不断的读取socket的数据
	std::uint32_t read_sz = tcp_socket.read_some(asio::buffer(buffer + buffer_begin, MAX_PACKET_SZ - buffer_begin));
	buffer_begin += read_sz;
	while(buffer_begin > sizeof(std::uint32_t)>)
	{
		// 当读取了头部的四个字节之后 我们就知道了当前packet的大小
		std::uint32_t cur_packet_sz = 0;
		std::copy(buffer, buffer + sizeof(std::uint32_t), reinterpret_cast<char*>(&cur_packet_sz));
		assert(cur_packet_sz + 4 < MAX_PACKET_SZ>);
		if(buffer_begin >= cur_packet_sz + sizeof(std::uint32_t))
		{
			// 如果当前buffer中的数据长度已经大于等于packet的所需大小 可以将此packet向业务层传递
			packet_callback(buffer + sizeof(std::uint32_t), cur_packet_sz);
			// 从buffer中删除这个已经处理的packet相关的数据
			std::copy(buffer + sizeof(std::uint32_t) + cur_packet_sz, buffer + buffer_begin, buffer);
			buffer_begin -= sizeof(std::uint32_t) + cur_packet_sz;
			// 这里会执行下一层的while去尝试处理下一个packet 因为当前buffer里面可能有多个packet
		}
		else
		{
			// 剩下的数据不足以拼装成一个packet 等待后续数据
			break;
		}
	}
}

上面的代码只是为了大概解释一下TCP接收端的解包过程，真正的生产环境代码不能直接这么写，主要有如下两个问题：

1. 这里预先设置了单Packet的最大大小为65536也就是64k，大于此大小的Packet将会触发Assert，但实际的业务逻辑中以M为单位的Packet也是可能有的。解决的方法有两种：采用动态大小的Buffer，或者发送端将大的业务包拆分为多个连续的合适大小小包，对应的接收端处理小包拼接逻辑。
2. 这里每次处理完一个Packet之后会将剩余的数据重新Copy到Buffer的开头，这里其实可以修改为只要剩余的Buffer能够容纳下一个完整的Packet就不需要执行Copy，这样就可以节省很多内存拷贝的时间。

游戏网络通信中的UDP

UDP与TCP之间的差异

前面一节所展示的客户端与服务器之间的通信使用的是基于TCP的Socket，TCP是建立于IP网络层协议之上一种面向连接的、可靠的、基于字节流的传输层协议。为了实现可靠性，TCP实现了流量拥塞控制功能。建立在IP网络层协议之上还有一种知名的传输层协议UDP，他是一种无连接的面向数据包的不可靠传输层协议。即UDP在发送数据之前不需要走类似于TCP的三次握手，只要知道目标的ip:port就可以直接调用sendto接口来发送一段二进制数据。然后sendto接口只负责尝试将这段数据发送到系统的发送缓冲区，如果缓冲区满了甚至可以无声无息的丢弃这段数据。然后系统底层为UDP实现的socket可读通知是保证一个完整的UDP包被接收了，即一次recvfrom一定等价于一次sendto。但是反过来一次sendto不一定等价于一次recvfrom，因为这个被发送的包数据可能在传输的过程中被丢弃。同时recvfrom的顺序不一定等于sendto的顺序，这样就会导致包的乱序到达。业务层使用recvfrom接收udp包的时候需要传入buf和bufsize，就是接收空间和接收空间大小。如果这个bufsize小于udp包的大小，那么只能接收到这个udp包的前bufsize个字节，剩下的部分会被直接被丢弃，再次执行recvfrom的时候处理的已经是第二个包了。所以bufsize要适配组包时的单Packet大小上限，一般来说这个大小上限都会设置为ip的MTU大小，这样避免单UDP包在传输的时候被拆分为多UDP包，多包引发丢包的概率比单包丢失概率大很多。

在互联网业务中网络通信使用的传输层协议基本全是TCP，主要是因为其可靠性可以减少上层逻辑复杂度。而在游戏业，随着手机游戏的发展，UDP协议在那些要求低延迟的游戏品类里逐渐成为了主流选择。因为手机游戏所处的网络环境比PC环境复杂的多，网络的接入主要是移动网络或者Wifi，这两种网络相对于PC游戏常用的有线网络来说稳定性降低了很多，随着手机物理位置的移动会随机的触发丢包。在使用TCP协议时，如果遇到丢包，则TCP协议会认为此时的网络信道出现了阻塞，因此会触发TCP协议的拥塞避免，此时会将TCP协议的发送窗口减半,同时发送速率降低到一个最大报文段Maximun Segment Size，开始慢启动流程。

如果出现连续的多个丢包，则会导致TCP协议进行多次减半发送窗口，这样会导致丢包后的发送速率急剧下降，服务器端收到的后续数据包的延迟急剧增大。在网络游戏中，客户端延迟是玩家游戏体验的极其重要的一环，在动作类和FPS类游戏中大于100ms的延迟会导致游戏体验基本为0！而随机性的延迟飙升更容易触发玩家的愤怒，引发各种恶评。丢包引发的延迟飙升是TCP的内在机制决定的，无法从业务逻辑层绕过。所以这类低延迟要求的游戏很多都从TCP切换到了UDP，因为UDP并没有带流量控制功能。但是切换到UDP又会导致TCP所带的数据可靠性丧失，因为UDP并没有TCP的ACK与超时重传机制。为了避免影响上层业务逻辑对网络的处理，一般来说这类游戏会基于UDP实现一个带ACK与超时重传机制的可靠UDP协议。

KCP:可靠UDP的一种实现

KCP是一个快速可靠协议，能以比 TCP浪费10%-20%的带宽的代价，换取平均延迟降低30%-40%，且最大延迟降低三倍的传输效果。纯算法实现，并不负责底层协议（如UDP）的收发。需要使用者自己定义下层数据的发送方式，以 callback的方式提供给 KCP。连时钟都需要外部传递进来，内部不会有任何一次系统调用。

KCP力求在保证可靠性的情况下提高传输速度。KCP没有规定下层传输协议，作为一个逻辑层协议它也可以运行在TCP之上。但通常使用UDP来实现，因为TCP自带的拥塞控制会导致上层的KCP丧失所有意义。 KCP内部通过如下几个机制来实现快速可靠：

1. RTO不翻倍。RTO(Retransmission-TimeOut)即重传超时时间，TCP的超时计算是RTO*2，而KCP的超时计算是RTO*1.5，也就是说假如连续丢同一个包3次，TCP第3次重传是RTO*8，而KCP则是RTO*3.375，意味着可以更快地重新传输数据。

2. 更优的ACK机制 TCP在连续ARQ（自动重传请求，Automatic Repeat-reQuest）协议中，不会将一连串的每个数据都响应一次，而是延迟发送ACK，通知对端此包之前的所有包都已经收到，目的是为了充分利用带宽，但是这样会计算出较大的RTT时间，延长了丢包时的判断过程。KCP在连续ACK的基础上，还可以对不连续的包进行ACK。KCP的ACK是否延迟发送可以调节，当配置了非延迟ACK时，收到数据立即响应。

3. 选择性重传 TCP中实现了连续ARQ协议，再配合累计确认重传数据，只不过重传时需要将最小序号丢失的以后所有的数据都要重传；而KCP能够对不连续的包进行ACK，这样发送端就可以单独对所有已发出但未ACK的数据单独做计数，故而只需要重传真正丢失的数据。

4. 非退让流控 TCP在发生丢包时会将发送窗口减半，但KCP不做处理，这样对其他做传输的服务是不公平的，如果网络真的拥堵，KCP如此将导致网络里增加更多的未被收到的数据（更多的丢包），牺牲了带宽利用率

KCP拥有上述多个优点，但是使用KCP实现一个可靠的UDP还是需要一些工作量的。

在应用层通过kcp_send 发送数据，KCP 会把用户数据拆分 KCP 数据包，通过 kcp_output 再以 UDP 包的方式发送出去。具体细节上可以拆分为如下几步：

1. 创建KCP对象，这里需要一个表示会话编号的整数conv，代表session的标识符，通信双方需要保证使用的标识符一致，这部分一般通过先在两端建立一个TCP连接，在这个TCP连接中商定好对应的conv之后，再创建对应的KCP对象 。创建的接口如下

	void* user;
	ikcpcb *kcp = ikcp_create(conv, user);

这里的user是一个void*，用来配合KCP的相关回调函数来使用。

2. 设置发送回调函数，作为KCP下层协议的输出函数，KCP需要发送数据时会调用此函数。

// buf/len 表示缓存和长度。
// user 指针为 kcp 对象创建时传入的值，用于区别多个 KCP 对象。
int udp_output(const char *buf, int len, ikcpcb *kcp, void *user) {
		....
}
// 设置回调函数。
ikcp_setoutput(kcp, udp_output);

3. 调用ikcp_send来发送数据，注意这里只是把数据放到KCP内部的发送缓冲区，不一定会触发之前设置好的发送回调

int ikcp_send(ikcpcb *kcp, const char *buffer, int len);

4. 循环调用ikcp_update，来更新KCP的内部状态，检查是否需要发送或者超时重传。

ikcp_update(kcp, millisec);

在应用层通过底层网络库提供的UDP接收数据功能收到任意数据之后，调用ikcp_input将接收的数据拷贝到KCP的内部接收缓冲区，然后调用ikcp_recv来检查是否接收到一个应用层可以处理的包。

ikcp_input(kcp, received_udp_packet, received_udp_size);
int recv_packet_sz = ikcp_recv(ikcpcb *kcp, char *buffer, int len);
if(recv_packet_sz > 0)
{
	on_recv(buffer, recv_packet_sz);
}

特别值得注意的是KCP并没有考虑线程安全，所以应用层需要自己处理对同一个KCP对应的多线程互斥访问。对于KCP连接的客户端来说，这种互斥访问可以很方便的使用asio::strand来解决。但是对于使用KCP的监听服务器而言，它同时维护着多个客户端连接对应的KCP对象，不仅需要考虑每个单独的KCP对象的多线程互斥访问，同时还要考虑多个KCP的并发数据发送问题。因为监听服务器只有一个asio::udp::socket对象，不像之前的TCP服务器对于每个连接都构造一个asio::tcp::socket对象。同一个socket对象上的read, send, async_read, async_send都是不能并行化的，同时一个异步操作完成之前不能再发起同类型异步操作。对于KCP服务器而言主要要处理的就是多个KCP对象的异步发送。所以这里推荐使用一个带std::mutex的线程安全队列来存储所有的发送请求。

class kcp_acceptor
{
	std::unordered_map<asio::ip::udp::endpoint, std::shared_ptr<kcp_socket_wrapper>> m_client_connections;
	std::queue<std::pair<asio::ip::udp::endpoint, std::shared_ptr<std::string>>> m_send_queues;
	std::mutex m_send_queue_lock;
	std::shared_ptr<asio::ip::udp::socket> m_socket;

	// 对外暴露的数据发送接口
	void do_send(asio::ip::udp::endpoint send_to_endpoint, std::string_view data)
	{
		auto temp_data = std::make_shared<std::string>(data.data(),data.size());
		std::lock_guard<std::mutex> temp_lock(m_send_queue_lock);
		m_send_queues.push(std::make_pair(send_to_endpoint, temp_data));
		if (m_send_queues.size() == 1)
		{
			auto cur_front = m_send_queues.front();
			m_socket->async_send_to(asio::buffer(*cur_front.second), cur_front.first, [this](const std::error_code&, size_t)
				{
					after_send();
				});
		}
	}
	void after_send()
	{
		std::shared_ptr<kcp_socket_wrapper> temp_kcp_wrapper;
		{
			std::lock_guard<std::mutex> temp_lock(m_send_queue_lock);
			if (m_send_queues.empty())
			{
				return;
			}
			
			auto cur_front_endpoint = m_send_queues.front().first;
			auto temp_iter = m_client_connections.find(cur_front_endpoint);
			if (temp_iter != m_client_connections.end())
			{
				temp_kcp_wrapper = temp_iter->second;
			}
		}
		if(temp_kcp_wrapper)
		{
			temp_kcp_wrapper->check_write_finish();
		}
		{
			std::lock_guard<std::mutex> temp_lock(m_send_queue_lock);
			m_send_queues.pop();
			if (m_send_queues.empty())
			{
				return;
			}
			auto cur_front = m_send_queues.front();
			m_socket->async_send_to(asio::buffer(*cur_front.second), cur_front.first, [this](const std::error_code&, size_t)
				{
					after_send();
				});
		}
		
	}
}

连接层加密

客户端和服务器之间进行网络通信时，有权访问网络的人员可以监视所有流量并检查客户端和服务器之间发送或接收的数据。如果连接之间传递的数据都是明文，则居心叵测的人员可以通过监听连接流量的方式来获取用户登录的账号信息以及通过解析数据包的形式来发送恶意指令并获利。为了避免明文传输数据，使得传输的数据不可读，我们需要将连接进行加密。

加密又分为两种：对称加密和非对称加密。对称加密里的加密和解密key是一样的，而非对称加密则分为公钥和私钥之分，两个密钥的内容不同，公钥公开给其他使用当前加密服务的人员使用，私钥则自己存储，一份数据经过公钥加密后可以通过私钥解密，同样的经过私钥加密之后可以通过公钥解密。这种公钥分享系统也叫做PKI即Public Key Infrastructure。

由于非对称加密的复杂度一般远远大于对称加密的复杂度， 所以实际使用时一般是首先通过公钥加密系统来握手，同时商定对称加密的key， 之后的处理都走对称加密流程。这样既确保了对称密钥的私密性，又加大了数据处理的速度。

具体实现上我们可以使用openssl这个加密库来进行流量的加密传输，非对称加密系统采取RSA， 对称加密系统则采用AES，服务提供者将RSA生成的公钥发布到网络，生成的私钥保留在服务器。此时客户端与服务器之间的通信流程参考浏览器访问https网页的通信流程：

Mosaic Game 的网络通信设计

连接管理

mosaic_game为了简化实现复杂度，在网络通信这里并没有考虑UDP，只支持了TCP的通信。同时为了支持上层逻辑中的断线重连与Service的动态迁移，构造出来了一个概念叫做anchor即锚点，作为整个mosaic_game的通信网络中的逻辑层通信通道来使用。底层的网络数据收发管理类是由asio::tcp::socket的封装net_connection作为基类:

class net_connection : public std::enable_shared_from_this<net_connection>
{
	asio::io_context& io;
	asio::ip::tcp::socket remote;
	std::shared_ptr<net_controller> net_controller_;
	block_t read_buffer;
	block_t send_buffer;
	std::shared_ptr<std::string> connection_name;
	std::string remote_endpoint_name;
};

但是业务层并不在乎消息的传递细节，消息的具体来源并不重要，所以业务收发消息的时候并不直接对接这个net_connection。桥接业务层的对象是net_controller，这个类型对接业务逻辑收发数据。

class net_controller
{
public:
	// 数据接收到时的回调 length为成功读取的数据大小 返回值为错误信息
	virtual std::string on_data_read(const unsigned char* data, std::size_t length) = 0;
	// 数据发送时的回调 length为发送成功的数据大小 返回值为错误信息
	virtual std::string on_data_send(const unsigned char* data, std::size_t length) = 0;
	// 尝试获取一个完整数据包，返回下一次读取数据时的最少数据大小
	virtual std::size_t data_should_read(std::size_t max_length) = 0;
	// 往buffer中填充发送数据，返回填充的数据大小
	virtual std::size_t data_should_send(unsigned char* buffer, std::size_t max_length) = 0;
	virtual ~net_controller()
	{
		
	}
};

这个net_controller是一个纯虚类型，声明了一些接口函数，目前的实例化类型是net_channel来中转网络层数据。net_channel主要实现机制是提供两个线程安全的消息队列output_channel与input_channel，分别作为发送消息队列与接收消息队列。这样业务层读写数据只需要读写这两个队列，而无需关心底层信道的实现。同时net_connection里包含net_controller的指针，数据收发时会通过这个指针来调用相关的接口函数，来读写这两个队列。

class net_channel :public net_controller
{
	struct packet_header
	{
		std::uint32_t total_sz; // 当前包的完整大小
		std::uint16_t packet_cmd; // 当前包的消息类型
		std::uint8_t from_name_sz; // 发送者的anchor长度
		std::uint8_t dest_name_sz;// 目的地的anchor长度
	};

protected:
	// 下面两个成员是线程安全队列
	std::shared_ptr<channel<con_msg_task>> input_channel; // 接收到的packet放入到这个队列 等待业务层处理
	std::shared_ptr<channel<msg_task>> output_channel; // 业务层传递过来的发送数据
	input_block_buffer input_buffer; // 未读取完的多个packet所在的缓冲区
	output_block_buffer output_buffer; // 未发送完的多个packet所在的缓冲区

	std::vector<msg_task> sending_tasks; // 当前正在发送的多个packet
	std::shared_ptr<net_connection> connection;
	std::uint64_t m_temp_data_send_length = 0;
public:
	net_channel(std::shared_ptr<net_connection> in_connection, std::shared_ptr<channel<con_msg_task>> in_input_channel, std::shared_ptr<channel<msg_task>> in_output_channel);
}

此外为了管理net_channel所需的各项资源，额外在这个net_channel之外再构造了一个connection_resource对象，用来存储这个网络连接所绑定线程安全输入输出队列：

struct connection_resource
{
	std::shared_ptr<net_channel> connection_controller;
	std::unordered_set<std::string> anchors;
	std::shared_ptr<channel<con_msg_task>> input_channel;
	std::shared_ptr<channel<msg_task>> output_channel;

};

这里的anchors存储了在这个物理连接上暴露出来的所有实体地址的集合。业务主线程负责对接这个connection_resource的类型为network_router，每次有新连接创建时会创建一个对应的connection_resource，创建这个连接的输入输出队列，以及对应的队列控制块net_channel:

connection_resource* network_router::create_connection_resource(std::shared_ptr<net_connection> con)
{
	auto cur_iter = m_connection_resources.find(con.get());
	if (cur_iter != m_connection_resources.end())
	{
		return cur_iter->second.get();
	}
	auto new_resource = std::make_unique<connection_resource>();
	auto result = new_resource.get();
	new_resource->input_channel = m_input_msg_queue;
	new_resource->output_channel = std::make_shared<mutex_channel<msg_task>>();
	auto connection_channel = std::make_shared<net_channel>(con, new_resource->input_channel, new_resource->output_channel);

	new_resource->connection_controller = connection_channel;
	m_connection_resources.emplace(con.get(), std::move(new_resource));
	if (!con->is_outbound_connection())
	{
		m_inbound_connections[con->inbound_connection_idx] = con.get();
	}
	return result;
}

为了在逻辑层维护这个net_channel，basic_stub需要感知到物理连接的建立和断开，要求这些事件发生时推送一些连接控制消息到basic_stub中处理：

enum class connection_ctrl_msg_type
{
	// 主动连接建立
	on_connect = 0,
	// 连接断裂
	on_disconnect,
	// 被动连接建立
	on_accepted,
};
struct connection_ctrl_msg
{
	connection_ctrl_msg_type type;
	std::shared_ptr<net_connection> connection;
};

void basic_stub::on_conn_ctrl_msg(const network::connection_ctrl_msg& msg)
{
	switch (msg.type)
	{
	case network::connection_ctrl_msg_type::on_connect:
		return on_connect(msg.connection);
	case network::connection_ctrl_msg_type::on_disconnect:
		return on_disconnected(msg.connection);
	case network::connection_ctrl_msg_type::on_accepted:
		return on_accepted(msg.connection);
	default:
		break;
	}
}

连接控制消息的添加与读取都通过mutex_channel<connection_ctrl_msg>这个线程安全队列来实现，避免可能出现的多线程问题：

mutex_channel<connection_ctrl_msg> m_conn_ctrl_msgs;

void network_router::push_ctrl_msg(std::shared_ptr<net_connection> conn, connection_ctrl_msg_type msg_type)
{
	connection_ctrl_msg cur_msg{ msg_type, conn };
	m_conn_ctrl_msgs.push_msg(std::move(cur_msg));
}

void network_router::poll_ctrl_msg(const conn_ctrl_msg_callback_t& msg_handler)
{
	std::array<connection_ctrl_msg, 10> temp_tasks;
	while (true)
	{
		auto cur_poll_size = m_conn_ctrl_msgs.pop_bulk_msg(temp_tasks.data(), temp_tasks.size());
		if (cur_poll_size == 0)
		{
			return;
		}
		for (std::size_t i = 0; i < cur_poll_size; i++)
		{
			msg_handler(temp_tasks[i]);
		}
	}
}

当物理连接被建立时，会添加on_connected或者on_accepted消息，具体的消息类型取决于自己是不是监听方：

void net_connection::on_connected()
{
	router->push_ctrl_msg(shared_from_this(), connection_ctrl_msg_type::on_connect);
	request_send_data();
	this->async_read_data(true);

}
void basic_stub::do_accept()
{
	auto cur_listen_socket = std::make_shared<asio::ip::tcp::socket>(m_io_context);

	m_asio_wrapper->m_acceptor.async_accept([this](const asio_error_code& error, asio::ip::tcp::socket socket)
		{
			if (!m_asio_wrapper->m_acceptor.is_open())
			{
				return;
			}
			if(error)
			{
				m_logger->error("async_accept with error {}", error.message());
				return;
			}
			anchor_endpoint remote_endpoint(socket.remote_endpoint().address().to_string(), socket.remote_endpoint().port(), endpoint_type::tcp);

			auto cur_connection_idx = ++m_inbound_connection_counter;
			std::shared_ptr<network::net_connection> connection;
			if (m_local_server.rsa_key.empty())
			{
				connection = network::net_connection::create(m_io_context, std::move(socket), cur_connection_idx, m_logger, m_connection_timeout, m_router.get());
			}
			else
			{
				connection = network::encrypt_connection::create(m_io_context, std::move(socket), cur_connection_idx, m_logger, m_connection_timeout, m_router.get(), m_local_server.rsa_key);
			}
			
			this->m_router->push_ctrl_msg(connection, network::connection_ctrl_msg_type::on_accepted);
			//connection->start_as_server();
			//this->router->accept_endpoint(connection);
			do_accept();
		});
}

这里的on_accepted对应的是当前进程作为监听服务器收到了一个入站连接，其逻辑很简单，开启对消息的接收：

void basic_stub::on_accepted(std::shared_ptr<network::net_connection> connection)
{
	connection->start_as_server();
	m_router->accept_endpoint(connection);
}

这里的accept_endpoint会为这个连接构造一个代理对象connection_resource，维护这个连接对应的收发队列input_channel与output_channel:

bool network_router::accept_endpoint(std::shared_ptr<net_connection> in_connection)
{
	auto resource_ptr = create_connection_resource(in_connection);
	in_connection->set_controller(resource_ptr->connection_controller);
	return true;
}

而on_connect刚好与on_accepted相反，代表本进程的一个出站连接被成功接收，这里的处理逻辑是开启这个连接的保活，定期发送一个心跳包：

void basic_stub::on_connect(std::shared_ptr<network::net_connection> connection)
{
	const auto& cur_connection_name = connection->get_connection_name();
	m_logger->info("on_connect for {}", *cur_connection_name);
	m_keep_alive_servers[*cur_connection_name] = connection;
}

void basic_stub::keep_alive_callback()
{
	std::string temp_keep_alive_anchors;
	for (const auto& one_server : m_keep_alive_servers)
	{
		if (one_server.second)
		{
			send_keep_alive(one_server.first);
			temp_keep_alive_anchors += one_server.first + " ";
		}
	}
	if(!temp_keep_alive_anchors.empty())
	{
		m_logger->debug("keep_alive_callback servers {}", temp_keep_alive_anchors);
	}
	
	add_timer_with_gap(std::chrono::milliseconds(m_connection_timeout / 2), [this]()
		{
			this->keep_alive_callback();
		});
}

当物理连接断开的时候，会往network_router里推送连接断开消息：

void net_connection::close_connection()
{
	if (stopped)
	{
		return;
	}
	stopped = true;
	this->cancel_all_timers();

	if (connection_name)
	{
		logger->error("close connection for {} {}", *connection_name, remote_endpoint_name);
	}
	else
	{
		logger->error("close connection for {} ", remote_endpoint_name);
	}
	asio_error_code ec;
	if (this->remote.is_open())
	{
		this->remote.shutdown(asio::ip::tcp::socket::shutdown_both, ec);
		this->remote.close(ec);
	}
	if (net_controller_)
	{
		net_controller_.reset();
	}
	router->push_ctrl_msg(shared_from_this(), connection_ctrl_msg_type::on_disconnect);
	return;
}

对应的回调处理on_disconnected则复杂一些，需要考虑一下异常断线还是正常断线，是否需要重新连接。

void basic_stub::on_disconnected(std::shared_ptr<network::net_connection> connection)
{
	m_router->disconnect(connection);

	const auto& cur_connection_name = connection->get_connection_name();
	if (!cur_connection_name || cur_connection_name->empty())
	{
		m_logger->info("empty conn name for {}", connection->get_remote_endpoint_name());
		return;
	}

	if (!should_reconnect(connection))
	{
		return;
	}

	// re connect
	m_logger->info("reconnect to server {} after {} ms", *cur_connection_name, m_timer_check_gap_ms);
	add_timer_with_gap(std::chrono::milliseconds(m_timer_check_gap_ms), [cur_connection_name, this]()
		{
			connect_to_server(*cur_connection_name);
		});
}

数据发送

net_channel中用一个packet_header的结构体来进行封包，成员变量packet_seq代表包的流水号，total_sz代表当前包的整体长度，packet_cmd代表当前包里业务数据的类型:

struct packet_header
{
	std::uint64_t packet_seq; // 当前包序列号 
	std::uint32_t total_sz; // 当前包的完整大小
	std::uint16_t packet_cmd; // 当前包的消息类型
	std::uint8_t from_name_sz; // 发送者的anchor长度
	std::uint8_t dest_name_sz;// 目的地的anchor长度
};

这里的packet_seq只有在客户端与服务器之间的数据包才会赋值，其他情况下默认为0，发送数据的时候network_router提供如下几个接口：

bool push_msg(const std::string& from, const std::string& dest, const std::string& data, std::uint16_t cmd);
void broadcast_msg(const std::string& from, const std::vector<std::string>& ids, const std::string& data, std::uint16_t cmd);

bool push_msg(const std::string& from, const std::string& dest, std::shared_ptr<const std::string> data, std::uint16_t cmd);
bool push_msg(std::shared_ptr<const std::string> from, std::shared_ptr<const std::string> dest, std::shared_ptr<const std::string> data , std::uint16_t cmd, msg_seq_t msg_seq = 0);
;
bool push_msg(const net_connection* connection, std::shared_ptr<const std::string> from, std::shared_ptr<const std::string> dest, std::shared_ptr<const std::string> data, std::uint16_t cmd, msg_seq_t msg_seq = 0);
bool push_msg(std::uint64_t inbound_con_idx, std::shared_ptr<const std::string> from, std::shared_ptr<const std::string> dest, std::shared_ptr<const std::string> data, std::uint16_t cmd);

这里的from to都是anchor，每个anchor最多绑定一个net_connection：

class anchor_resource
{
	friend class anchor_collection;
public:
	const std::string name;
private:
	const net_connection* connection;

	channel<msg_task> output_channel;
};

往一个anchor发送消息执行逻辑分为两种情况：

1. 如果有绑定的net_connection, 就是往对应的net_connection的output_channel里添加队尾数据。
2. 如果没有绑定的net_connection，则往自身的output_channel里添加队尾数据

当一个anchor绑定到net_connection时，将output_channel里的数据拼接到net_connection的output_channel中。当一个net_connection断开时，按序取出output_channel中的所有数据，投递到对应的anchor_resource内部的output_channel中。在这样的设计下，物理连接的断开不至于导致发送消息的丢失。这样的机制下可以保证一个anchor切换连接时，其数据的发送仍然保持可靠且有序。

bool network_router::push_msg(std::shared_ptr<const std::string> from, std::shared_ptr<const std::string> dest, std::shared_ptr<const std::string> data, std::uint16_t cmd)
{
	if (!dest)
	{
		dest = m_empty_dest;
	}
	if(!from)
	{
		from = m_empty_dest;
	}
	if(dest->rfind(m_local_anchor_name, 0) == 0)
	{
		// 说明是本进程地址 直接推送数据到input_msg_queue
		network::con_msg_task local_msg_task;
		local_msg_task.first = {};
		local_msg_task.second = msg_task::construct(from, dest, data, cmd);
		m_input_msg_queue->push_msg(std::move(local_msg_task));
	}
	auto cur_proxy_resource = m_anchor_collection.find_proxy_for_anchor(*dest);
	if(!cur_proxy_resource)
	{
		m_logger->error("push_msg cant find anchor_resources from {} dest {}  data {}", *from, *dest, *data);

		return false;
	}
	auto cur_proxy_con = cur_proxy_resource->get_connection();
	if (cur_proxy_con)
	{
		if (push_msg(cur_proxy_con, from, dest, data, cmd))
		{
			return true;
		}
	}
	return cur_proxy_resource->try_push(from, dest, data, cmd);

}

上述的消息发送接口并不直接把发送数据打包为packet_header，而是封装为一个中间类型msg_task,添加到对应的发送队列中:

// bool network_router::push_msg(const net_connection* connection, std::shared_ptr<const std::string> from, std::shared_ptr<const std::string> dest, std::shared_ptr<const std::string> data, std::uint16_t cmd)
cur_connection_resource_iter->second->output_channel->push_msg(msg_task::construct(from, dest, data, cmd));


using msg_seq_t = std::uint64_t;
template <typename T, typename U>
struct channel_task
{
	std::shared_ptr<const T> data;
	std::shared_ptr<const U> dest;
	std::shared_ptr<const U> from;
	std::uint16_t cmd;
};
using msg_task = channel_task<std::string, std::string>;

这个类型的存在可以更好的维护发送与接受数据的生命周期，同时构造出逻辑层的消息包概念。但是真正执行消息发送的时候我们需要将这个msg_task转换到之前定义好的packet_header规定的格式。由于TCP是一个数据流协议，所以每次发送的时候可以发送多个包数据，为此我们提供了一个缓冲区结构output_block_buffer来保存正在发送的多个包数据。底层网络连接在被建立时，会立即调用request_send_data来尝试填充缓冲区并发送：

void net_connection::on_connected()
{
	router->push_ctrl_msg(shared_from_this(), connection_ctrl_msg_type::on_connect);
	request_send_data();
	this->async_read_data(true);

}

void net_connection::request_send_data()
{
	if (net_controller_)
	{
		auto send_msg_sz = net_controller_->data_should_send(send_buffer.data(), send_buffer.size());
		if (send_msg_sz)
		{
			async_send_data(send_msg_sz);
			return;
		}
	}
	cancel_timer(timer_type::check_send);

	set_timer(timer_type::check_send, std::chrono::milliseconds(5));
}

这里的net_channel::data_should_send来不断的获取发送队列中的msg_task来填充output_buffer对应的数据缓冲区，然后将相关数据复制到真正的发送缓冲区buffer，直到发送队列为空或者缓冲区已经满了。此函数内部会进行packet_header封包以及按buffer::max_length进行拆包。

std::size_t net_channel::data_should_send(unsigned char* buffer, std::size_t max_length)
{
	
	std::size_t consume_sz = 0;
	// 从output_buffer中提取剩下的数据到buffer中
	consume_sz += output_buffer.consume(buffer + consume_sz, max_length - consume_sz);
	channel_task<std::string, std::string> temp_task;
	while(consume_sz < max_length)
	{
		// 如果buffer没有满 则尝试从channel中获取下一个发送的task
		if (!output_channel->pop_msg(temp_task))
		{
			break;
		}
		packet_header header;
		header.dest_name_sz = std::uint8_t(temp_task.dest->size());
		header.from_name_sz = std::uint8_t(temp_task.from->size());
		header.packet_cmd = temp_task.cmd;
		header.total_sz = sizeof(packet_header) + header.from_name_sz + header.dest_name_sz + static_cast<std::uint32_t>(temp_task.data->size());
		//std::cout << "cur packet sz is " << header.total_sz << std::endl;

		auto header_ptr = std::make_shared<std::string>(reinterpret_cast<const char*>(&header), sizeof(packet_header));
		output_buffer.add(header_ptr);
		output_buffer.add(temp_task.from);
		output_buffer.add(temp_task.dest);
		output_buffer.add(temp_task.data);
		// 将新的发送task打包好之后放入到output_buffer中 然后继续填充buffer
		consume_sz += output_buffer.consume(buffer + consume_sz, max_length - consume_sz);
		m_sending_tasks.push_back(temp_task);
		m_packet_send_counter++;
		m_data_send_sz += header.total_sz;
	}
	return consume_sz;
}

这里的sending_task记录了当前正在output_buffer缓冲区内packet，这个数组的存在是为了支持后续的断线重连操作，

void net_connection::async_send_data(std::size_t size)
{
	logger->trace("async_send_data with data size {}",  size);
	async_send_data_impl(0, size);
}
void net_connection::async_send_data_impl(std::size_t offset,  std::size_t total_size)
{
	auto self(this->shared_from_this());
	this->set_timer(timer_type::send, timeout);

	this->remote.async_write_some(asio::buffer(send_buffer.data() + offset, total_size - offset),
		asio::bind_executor(this->strand, [this, self, offset, total_size](const asio_error_code& error, std::size_t bytes_transferred)
	{
		if (this->cancel_timer(timer_type::send))
		{
			if (!error)
			{
				on_data_send(bytes_transferred);
				if (bytes_transferred + offset  < total_size)
				{
					logger->trace("send with bytes transferred {}", bytes_transferred);
					this->async_send_data_impl(offset + bytes_transferred, total_size);
				}
				else
				{
					on_buffer_all_send(total_size);
				}
			}
			else
			{
				logger->warn("report error at {}", "async_send_data_impl");
				this->on_error(error);
			}
		}
	})
	);
}
void net_connection::on_data_send(std::size_t bytes_transferred)
{
	if (net_controller_)
	{
		auto control_msg = net_controller_->on_data_send(send_buffer.data(), bytes_transferred);
		if (!control_msg.empty())
		{
			report_error(fmt::format("on_data_send error with msg {}", control_msg));
			return;
		}
	}
}

这里的net_connection::async_send_data_impl负责逐渐的发送缓冲区的数据，每次发送若干字节都会调用net_connection::on_data_send来通知network_channel::on_data_send，当缓冲区里的所有数据都发送完成之后，通过net_connection::on_buffer_all_send去从network_channel中获取新数据来填充net_connection的buffer:

void net_connection::on_buffer_all_send(std::size_t total_size)
{
	logger->trace("on_buffer_all_send  with size {}", total_size);
	request_send_data();
}

void net_connection::request_send_data()
{
	if (net_controller_)
	{
		auto send_msg_sz = net_controller_->data_should_send(send_buffer.data(), send_buffer.size());
		if (send_msg_sz)
		{
			async_send_data(send_msg_sz);
			return;
		}
	}
	cancel_timer(timer_type::check_send);

	set_timer(timer_type::check_send, std::chrono::milliseconds(5));
}

void net_connection::on_timeout(timer_type cur_timer)
{
	if (cur_timer == timer_type::check_send)
	{
		return request_send_data();
	}
	else if (cur_timer == timer_type::check_read)
	{
		return request_read_data();
	}
	logger->error("on_timeout for timer {}", timer_type_to_string(cur_timer));
	report_error( fmt::format("on_timeout for timer {}", timer_type_to_string(cur_timer)));
	return;
}

如果request_send_data里发现network_channel中的发送队列已经空了，则会开启一个计时器来定期检查发送队列里是否有新数据的到来。这里的开启timer来检查新数据是一个偷懒的方法，正确的方式应该是给这个network_channel设置一个队列增加新数据的回调，但是由于这里是多线程操作，实现一个正确的多线程通知接口可能有点复杂，所以加一个计时器最省事，代价就是新数据的发送可能有5ms的延迟。

数据接收

对应的解包逻辑在net_channel::on_data_read里，每收到一个完整的packet都会将这个packet放到input_channel的末尾:

void net_channel::on_data_read(const unsigned char* data, std::size_t length)
{
	input_buffer.add(data, length);
	while (true)
	{
		// 如果读取到的数据小于packet_header的长度 则等待下一次填充数据
		if (input_buffer.total_size() < sizeof(packet_header))
		{
			break;
		}
		std::uint32_t total_sz = 0;
		input_buffer.consume(reinterpret_cast<std::uint8_t*>(&total_sz), sizeof(std::uint32_t), true);
		// 获取当前packet的长度
		if (total_sz <= input_buffer.total_size())
		{
			// 如果读取到的数据大小大于当前packet的总长度 则代表接收到了一个包 开始解包
			channel_task<std::string, std::string> cur_task;
			packet_header cur_header;
			input_buffer.consume(reinterpret_cast<std::uint8_t*>(&cur_header), sizeof(packet_header), false);
			std::uint32_t cur_data_sz = total_sz - sizeof(packet_header) - cur_header.from_name_sz - cur_header.dest_name_sz;
			auto from_str = std::make_shared<std::string>(cur_header.from_name_sz, '0');
			input_buffer.consume(reinterpret_cast<std::uint8_t*>(from_str->data()), cur_header.from_name_sz, false);
			auto dest_str = std::make_shared<std::string>(cur_header.dest_name_sz, '0');
			input_buffer.consume(reinterpret_cast<std::uint8_t*>(dest_str->data()), cur_header.dest_name_sz, false);
			auto data_str = std::make_shared<std::string>(cur_data_sz, '0');
			input_buffer.consume(reinterpret_cast<std::uint8_t*>(data_str->data()), cur_data_sz, false);
			cur_task.data = std::move(data_str);
			cur_task.from = std::move(from_str);
			cur_task.dest = std::move(dest_str);
			cur_task.cmd = cur_header.packet_cmd;
			m_data_read_sz += cur_header.total_sz;
			m_packet_last_read_seq = cur_task.msg_seq;
			// 接收到的包发送到input_channel等待业务层轮询
			input_channel->push_msg(std::make_pair(connection, cur_task));
			// 执行下一次while 因为可能已经接收到了多个包
		}
		else
		{
			break;
		}
	}
}

这里所有的net_controller的input_channel指向的都是network_router里的m_input_msg_queue，这样就方便业务层去处理接收到的数据:

std::size_t network_router::poll_msg(const msg_handle_callback_t& msg_handler)
{
	std::array<con_msg_task, 10> temp_tasks;
	std::size_t result = 0;
	while (true)
	{
		auto cur_poll_size = m_input_msg_queue->pop_bulk_msg(temp_tasks.data(), temp_tasks.size());
		result += cur_poll_size;
		if (cur_poll_size == 0)
		{
			return result;
		}
		for (std::size_t i = 0; i < cur_poll_size; i++)
		{
			if (!msg_handler(temp_tasks[i].first, temp_tasks[i].second))
			{
				m_logger->error("fail to handler msg dest {}  cmd {} info {}", *(temp_tasks[i].second.dest),   temp_tasks[i].second.cmd, temp_tasks[i].second.data->substr(0, 50));
			}
			temp_tasks[i].second.clear();
			temp_tasks[i].first.reset();

		}
	}
}

然后在进程的主循环里会传递业务方自己的消息handler进去来处理消息:

void basic_stub::main_loop()
{
	m_logger->flush();
	auto cur_msg_handler = [this](std::shared_ptr<network::net_connection> con, const network::msg_task& one_msg)
	{
		return this->on_msg(con, one_msg);
	};
	auto cur_http_handler = [this](const http_utils::request& req, msg_seq_t req_seq)
	{
		return this->on_http_request(req, req_seq);
	};

	auto cur_conn_ctrl_msg_handler = [this](const network::connection_ctrl_msg& msg)
	{
		return this->on_conn_ctrl_msg(msg);
	};
	do
	{
		on_new_frame();
		poll_mainloop_tasks();
		auto poll_begin_ts = utility::timer_manager::now_ts();
		m_router->poll_msg(cur_msg_handler);
		http::http_request_mgr::poll_request(cur_http_handler);
		poll_timers(utility::timer_manager::now_ts());
		m_router->poll_ctrl_msg(cur_conn_ctrl_msg_handler);
		auto poll_end_ts = utility::timer_manager::now_ts();

		if (poll_end_ts - poll_begin_ts > m_high_load_threshold * m_timer_check_gap_ms)
		{
			continue;
		}
		else
		{
			break;
			
		}

	}while (!m_stopped);
}

以这样的形式来处理已经完整接受的消息数据，以线程安全队列的形式来保证消息已经在主线程处理，可以大大的减少逻辑编程的难度。

在basic_stub的类型定义中，on_msg函数被声明为了一个纯虚类，因为在basic_stub中并没有提供进程间通信的消息编码格式。真正的业务消息分发是在basic_stub的直接子类json_stub以及json_stub的后续子类之中。在json_stub中，给出了一个非常基础的消息格式定义。开头是一个uint16格式的消息包类型说明符，由两个uint8拼接而成，其中第一个uint8由全局枚举类型packet_cmd控制，第二个uint8的意义则完全依赖与第一个uint8的值。当前mosaic_game的packet_cmd中提供了七种消息类型：

enum class packet_cmd: std::uint8_t
{
	server_control = 0, // 进程控制消息
	client_to_game, // 客户端向服务端发消息
	game_to_client, // 服务端向客户端发消息
	server_rpc_msg, // 服务器之间的json rpc消息
	server_raw_msg, // 服务器之间的非json 消息
	entity_msg, 	// 发往场景进程里entity的消息
	actor_migrate_msg, // actor的迁移控制消息
	max,
};

为了方便的对这个消息包类型两个uint8的拼接与解析，提供了两个辅助函数：

struct packet_cmd_helper
{
	static std::uint16_t encode(packet_cmd in_packet_cmd, std::uint8_t in_cmd_detail)
	{
		std::uint16_t result = std::uint16_t(in_packet_cmd);
		result <<= 8;
		result += in_cmd_detail;
		return result;
	}

	static std::pair<packet_cmd, std::uint8_t> decode(std::uint16_t in_combine_cmd)
	{
		std::pair<packet_cmd, std::uint8_t> result;
		result.second = in_combine_cmd % 256;
		result.first = packet_cmd(in_combine_cmd / 256);
		return result;
	}
};

在basic_stub的直接基类json_stub中，只提供了最基础的server_control类型消息的分发，用来处理进程之间的相互注册：

bool json_stub::on_msg(std::shared_ptr<network::net_connection> con, const network::msg_task& one_msg)
{
	
	auto cur_cmd_detail = enums::packet_cmd_helper::decode(one_msg.cmd);
	switch (cur_cmd_detail.first)
	{
	case enums::packet_cmd::server_control:
		parse_and_dispatch_server_control_msg(con, one_msg);
		return true;
		break;

	default:
		return false;
	}
}

这里的parse_and_dispatch_server_control_msg就是将msg_task里的data字符串解析为一个json::object，内部包含一个类型为字符串的cmd字段和一个类型为json::object的param字段。解析完成之后再调用on_server_control_msg来处理进程角色的增删改查指令：

bool json_stub::on_server_control_msg(std::shared_ptr<network::net_connection> con, std::shared_ptr<const std::string> from,const std::string& cmd, const json& msg)

至于其他类型的消息处理，将放到RPC相关章节进行介绍。

在这个on_server_control_msg函数中处理了如下六种进程管理指令：

1. set_stub_info，将当前网络连接关联一个进程角色，这个指令都是在连接发起者在连接建立之后的json_stub::on_connect函数中发出的
2. query_stub_info，查询一个指定名字的进程角色信息，或者指定upstream的下游角色信息列表，对应的返回消息为reply_query_stub_info
3. send_keep_alive, 进程之间的心跳处理，由于连接长时间没有接收到消息会认为此连接已经断开，所以连接发起者会定期的发送这个心跳包来保活，对应的返回消息为reply_send_keep_alive
4. remove_stub_info，通知删除一个进程角色，一般是mgr_server通知一个进程的下线时对全服务器集群进行广播
5. notify_stop，通知当前进程启动关服流程
6. notify_clear_connection，关服流程中的一个子流程，通知所有的space_server与service_server开始断开所有的网络连接

在不同的json_stub的子类中，还可以继续拓展on_server_control_msg接受的命令类型，例如在redis_server中就额外支持了redis_request类型：

bool redis_server::on_server_control_msg(std::shared_ptr<network::net_connection> con, std::shared_ptr<const std::string> from, const std::string& cmd, const json& msg)
{
	if(json_stub::on_server_control_msg(con, from, cmd, msg))
	{
		return true;
	}
	if (cmd == "redis_request")
	{
		std::string errcode;
		redis::redis_task_desc cur_task_desc;
		std::uint64_t callback_id;
		do
		{
			try
			{
				msg.at("callback_id").get_to(callback_id);
				msg.at("request_detail").get_to(cur_task_desc);
			}
			catch (std::exception &e)
			{
				errcode = "invalid redis request msg";
				m_logger->error("on_server_control_msg fail to parse {} error {}", msg.dump(), e.what());
				break;
			}
		} while (0);
		// 这里省略了具体的消息处理逻辑
	}
	return false;
}

数据加密

为了避免客户端与服务器之间的消息协议被外挂人员轻松解析，客户端与服务器之间的消息基本都会经过加密之后再进行传输。在MosaicGame中，同样提供了加密连接的功能，在basic_stub::do_accept函数中，如果发现自己配置了rsa_key，则会将传入连接升级为加密连接encrypt_connection，否则就是普通的非加密连接net_connection:

void basic_stub::do_accept()
{
	auto cur_listen_socket = std::make_shared<asio::ip::tcp::socket>(m_io_context);

	m_asio_wrapper->m_acceptor.async_accept([this](const asio_error_code& error, asio::ip::tcp::socket socket)
		{
			if (!m_asio_wrapper->m_acceptor.is_open())
			{
				return;
			}
			if(error)
			{
				m_logger->error("async_accept with error {}", error.message());
				return;
			}
			anchor_endpoint remote_endpoint(socket.remote_endpoint().address().to_string(), socket.remote_endpoint().port(), endpoint_type::tcp);

			auto cur_connection_idx = ++m_inbound_connection_counter;
			std::shared_ptr<network::net_connection> connection;
			if (m_local_server.rsa_key.empty())
			{
				connection = network::net_connection::create(m_io_context, std::move(socket), cur_connection_idx, m_logger, m_connection_timeout, m_router.get());
			}
			else
			{
				connection = network::encrypt_connection::create(m_io_context, std::move(socket), cur_connection_idx, m_logger, m_connection_timeout, m_router.get(), m_local_server.rsa_key);
			}
			
			this->m_router->push_ctrl_msg(connection, network::connection_ctrl_msg_type::on_accepted);
			//connection->start_as_server();
			//this->router->accept_endpoint(connection);
			do_accept();
		});
}

对应的在主动发起连接的一方会根据目标服务器是否配置了rsa_key来选择使用加密连接还是普通连接，这样就能保证连接的双方同时启用加密连接或者普通连接：

bool network_router::connect_endpoint(const std::string& name, const anchor_endpoint& endpoint, const std::string& rsa_key)
{
	std::string log_key = fmt::format("{} {}:{}", name, endpoint.host, int(endpoint.port));
	auto connection_logger = utility::get_logger(log_key);
	if (m_named_connection.find(name) != m_named_connection.end())
	{
		m_logger->error("connection for {} already exist", name);
		return false;
	}
	m_logger->info("get logger for {}", log_key);
	connection_logger->info("begin connect_endpoint");
	std::shared_ptr<network::net_connection> cur_connection;
	if (rsa_key.empty())
	{
		cur_connection = net_connection::create(m_io_context, asio::ip::tcp::socket(m_io_context), 0, connection_logger, m_timeout, this);
	}
	else
	{
		cur_connection = encrypt_connection::create(m_io_context, asio::ip::tcp::socket(m_io_context), 0, connection_logger, m_timeout, this, rsa_key);
	}
	// 省略一些代码
}

由于只有客户端与服务器之间的通信有这个加密的需求，同时gate_server开启的监听端口只有客户端才会连接过来，所以只需要在gate_server的配置文件里写入这个rsa_key即可开启客户端与服务器之间的通信加密，同时保持服务器之间的通信仍然是明文的，避免加密带来的一些性能损耗。

在前文中我们曾经提到：基于RSA的非对称加密是非常消耗性能的，而基于AES的对称加密在性能上有非常大的优势，实际的加密系统中都是先用RSA加密的数据来执行握手并初始化两者之间的AES加密密钥，初始化完成之后后续的数据将只使用AES加密。在MosaicGame中也是这样设计的，客户端在创建encrypt_connection的时候，就会选择一种AES加密密钥，：

std::shared_ptr<encrypt_connection> encrypt_connection::create(asio::io_context& in_io, asio::ip::tcp::socket&& _in_remote_socket, std::uint64_t inbound_con_idx, std::shared_ptr<spdlog::logger> logger, std::chrono::milliseconds _in_timeout, network_router* in_router, const std::string& in_ras_key)
{
	auto new_connection = std::make_shared< encrypt_connection>(in_io, std::move(_in_remote_socket), inbound_con_idx, logger, _in_timeout, in_router, in_ras_key);
	if (new_connection->is_outbound_connection() && !new_connection->init_cipher("aes-256-cfb"))
	{
		logger->error("fail to init cipher");
		return {};
	}
	return new_connection;
}

在encrypt_connection中我们提供了十几种基于openssl的AES加密接口，这里偷懒就直接选择了aes-256-cfb这个加密方法。这里的init_cipher会根据指定的加密算法来初始化一个基础的加密密钥，并加上一个公用的前缀cipher_prefix，经过RSA加密之后存储到encrypted_cipher_info中：

std::string encrypt_connection::cipher_prefix()
{
	return "mosaic_game_cipher";
}
bool encrypt_connection::init_cipher(const std::string& cipher_name)
{
	std::string cipher_info_raw;

	cipher_info_raw += cipher_prefix();
	char cipher_code = 0;
	std::vector<unsigned char> ivec(16);
	std::vector<unsigned char> key_vec;
	if (cipher_name.size() > 7 && std::equal(cipher_name.begin(), cipher_name.begin() + 3, "aes"))
	{
		// aes
		encrypt::aes_generator::generate(cipher_name, cipher_code, ivec, key_vec, encryptor, decryptor);
	}

	if (!encryptor || !decryptor)
	{
		logger->error("failt to encrypt::aes_generator::generate");
		return false;
	}

	// 5 cipher code
	cipher_info_raw.append(1, static_cast<char>(cipher_code));
	cipher_info_raw.append(reinterpret_cast<char*>(ivec.data()), ivec.size());
	cipher_info_raw.append(reinterpret_cast<char*>(key_vec.data()), key_vec.size());

	std::array<unsigned char, 128> cipher_info_packed;
	if (cipher_info_raw.size() >= 128)
	{
		logger->error("cipher_info_raw.size() >= 128");
		return false;
	}
	
	std::copy(cipher_info_raw.begin(), cipher_info_raw.end(), cipher_info_packed.data());
	if (rsa_key.modulus_size() < 128)
	{
		logger->warn("invalid rsa public key");
		return false;
	}

	encrypted_cipher_info.resize(rsa_key.modulus_size());
	if (int(encrypted_cipher_info.size()) != rsa_key.encrypt(static_cast<std::uint32_t>(cipher_info_packed.size()), reinterpret_cast<unsigned char*>(cipher_info_packed.data()), encrypted_cipher_info.data(), encrypt::rsa_padding::pkcs1_oaep_padding))
	{
		logger->warn("invalid rsa encrypt size");
		return false;
	}
	return true;
}

随后在成功的与远端建立连接之后，在on_connected回调中会首先将这个encrypted_cipher_info发送到对端，同时开启对指定的密钥确认数据的等待：

void encrypt_connection::on_connected()
{
	std::copy(encrypted_cipher_info.begin(), encrypted_cipher_info.end(), send_buffer.data());
	this->async_send_data(this->encrypted_cipher_info.size());
	this->async_read_data(true, cipher_accept_reply_info().size());
}

encrypt_connection的接收端在启动的时候会首先设置当前连接的状态为等待RSA握手状态，然后启动对握手密钥的读取,由于我们在客户端连接中已经将握手数据扩张为了rsa_key.modulus_size()大小，所以这里会等待指定字节的数据到达：

void encrypt_connection::start_as_server()
{
	encrypt_key_send = true;
	encrypt_key_accepted = false;
	async_read_data(true, rsa_key.modulus_size());
}

当数据到达之后，如果自己是加密连接的接收端，会检查握手是否已经完成，如果没有完成则检查传入的数据是否是合法的握手数据，如果是则使用指定的数据来初始化AES加密密钥,同时将密钥确认信息发送回远端：

std::string encrypt_connection::cipher_accept_reply_info()
{
	return "entity mesh cipher accepted";
}
void encrypt_connection::on_data_read(std::size_t bytes_transferred)
{
	logger->trace("encrypt_connection::on_data_read bytes_transferred {}", bytes_transferred);
	if (!is_outbound_connection())
	{
		if (!this->encryptor)
		{
			if (!accept_cipher(read_buffer.data(), bytes_transferred))
			{
				report_error("accept_cipher");
				return;
			}
			logger->info("accept_cipher suc for client {}:{}", remote.remote_endpoint().address().to_string(), remote.remote_endpoint().port());
			auto cipher_ack_msg = cipher_accept_reply_info();
			encryptor->encrypt(reinterpret_cast<unsigned char*>(cipher_ack_msg.data()), send_buffer.data(), cipher_ack_msg.size());
			async_send_data(cipher_ack_msg.size());
			async_read_data();
			return;
		}
	}
	// 暂时省略后续代码

}

由于accept_cipher会初始化encryptor，开启后续数据的加密，所以这里的cipher_ack_msg也会经过这样的加密，然后再发送回发起端。由于发送端在创建初始密钥的时候已经用这个密钥创建了一个解密的decryptor，所以接受来的消息都需要经过decryptor进行解密。由于我们采用的是AES的流式加密算法，加密前后的字节数量永远相等，所以发起端的on_data_read第一次回调的时候，基本可以保证是握手成功的消息：

void encrypt_connection::on_data_read(std::size_t bytes_transferred)
{
	logger->trace("encrypt_connection::on_data_read bytes_transferred {}", bytes_transferred);
	// 省略前面介绍了的接收端握手处理
	decryptor->decrypt(read_buffer.data(), decrypt_buffer.data(), bytes_transferred);
	std::copy(decrypt_buffer.data(), decrypt_buffer.data() + bytes_transferred, read_buffer.data());
	if (!encrypt_key_accepted)
	{
		encrypt_key_accepted = true;
		auto accept_str = cipher_accept_reply_info();
		if (bytes_transferred != accept_str.size())
		{
			logger->error("expect accept_str {} while size {} mot match ", accept_str, bytes_transferred);
			return;
		}
		for (std::size_t i = 0; i < bytes_transferred; i++)
		{
			if (read_buffer[i] != accept_str[i])
			{
				logger->error("expect accept_str {} while received {}", accept_str, std::string(read_buffer.data(), read_buffer.data() + bytes_transferred));
				return;
			}
		}
		router->push_ctrl_msg(shared_from_this(), connection_ctrl_msg_type::on_connect);
		request_send_data();
		async_read_data();

		return;
	}
	net_connection::on_data_read(bytes_transferred);

}

当确认了收到的消息是握手成功消息之后，连接建立的真正回调才会推送到业务主线程，同时通过request_send_data开始读取业务消息推送队列，不过这里填充了发送数据的buffer之后，还需要通过encryptor->encrypt执行一次加密：

void encrypt_connection::request_send_data()
{
	if (net_controller_)
	{
		auto send_msg_sz = net_controller_->data_should_send(encrypt_buffer.data(), encrypt_buffer.size());
		if (send_msg_sz)
		{
			encryptor->encrypt(encrypt_buffer.data(), send_buffer.data(), send_msg_sz);
			async_send_data(send_msg_sz);
			return;
		}
	}
	else
	{
		logger->warn("encrypt_connection::request_send_data net_controller not set");
	}
	set_timer(timer_type::check_send, std::chrono::milliseconds(5));
}

此时客户端与服务器的连接握手过程完整结束，后续数据的读取与发送都会通知到对应的net_controller，两端发送数据的时候都会执行AES加密，两端接收数据的时候都会执行AES解密。由于加密和解密这两个过程都能保证前后的字节数量一摸一样，因此断线重连的相关流程也不需要修改，因为一个完整的packet所占用的字节数量在加密之后是一样的，这样packet_header里记录的total_sz仍然是正确的。

断线重连

为了避免客户端由于弱网和切换网络导致的断线引发游戏重新登录这种不良体验，我们在gate_server层做了客户端断线重连的逻辑。断线重连的核心就是维护一个session会话,在同一个session内的gate_server发往client的所有数据包的packet_seq是递增的，客户端记录自己业务层已经接收到的最大packet_seq，在断线重连的时候告知gate_server这个最大已接收packet_seq，然后gate_server将对序列号大于此packet_seq的消息包进行重新传输，这样就保证了重连之后客户端接收到的消息包packet_seq永远是连续递增的，不会出现丢失与重复。至于client发往gate_server的数据则不需要做这样的可靠性保证，因为客户端发送到服务器的请求总是可以重试的。

当物理链路断开时，net_channel需要将未发送完成的packet重新插入到output_channel的头部，以保证消息不丢失以及有序。为此我们需要记录那些还没有发送成功的消息。之前我们在net_channel::data_shoud_send函数中会将业务层推送过来的消息按顺序填充到发送缓冲区output_buffer，然后再由net_connection::async_send来将这个发送缓冲区里的数据进行发送。由于TCP是流式协议，填充到发送缓冲区的数据并不保证一次性全量发送完成，如果这个缓冲区数据有多个逻辑packet，可能会出现远端已经接收了开头的若干packet的情况。为了避免这些已经被接收的packet在断线重连之后被重新发送，在一个业务层消息被填充到发送缓冲区的时候我们将这个消息填充到正在发送队列sending_tasks的末尾。同时在网络连接这边需要在每次发送若干字节后都调用这里提供的on_data_send方法，这个函数负责检查一个或多个packet是否已经发送成功，如果发送成功就从sending_tasks队列的头部弹出。这里还有个特别需要注意的地方，async_send发送成功并不代表TCP连接的对端接收成功，只是代表数据已经转移到了系统提供的发送缓冲区。当真正的断线发生的时候，这个残留在发送缓冲区里的数据仍然是没有被对端收到的，因此如果要做一个正确的断线重连，我们还需要保留一些async_send发送成功的消息包到发送完成队列m_already_send_tasks中，这个保留的任务最大数量由m_already_send_task_max_num变量来控制, 其实更好的方法是通过时间戳来控制：

std::string net_channel::on_data_send(const unsigned char* data, std::size_t length)
{
	(void)data;
	m_temp_data_send_length += length;
	while (m_temp_data_send_length)
	{
		if (sending_tasks.empty())
		{
			return "wtf sending_tasks.empty()";
		}
		const auto& temp_task = sending_tasks.front();
		std::size_t total_sz = sizeof(packet_header) + temp_task.from->size() + temp_task.dest->size() + temp_task.data->size();
		//std::cout<<"total_sz "<<total_sz<<" length "<<length<<" task count "<<sending_tasks.size()<<std::endl;
		if (total_sz <= m_temp_data_send_length)
		{
			// 如果开启了断线重连的支持 则将已发送数据包添加到已发送队列的末尾
			// 通知控制队列最大元素数量
			if(m_already_send_task_max_num)
			{
				m_already_send_tasks.push_back(temp_task);
				if(m_already_send_tasks.size() > m_already_send_task_max_num)
				{
					m_already_send_tasks.pop_front();
				}
			}
			sending_tasks.erase(sending_tasks.begin());
			m_temp_data_send_length -= total_sz;
		}
		else
		{
			break;
		}
	}
	return std::string();
}

所以在断线时，不仅要将正在发送队列sending_tasks里的消息包重新放回output_channel，还要把已发送数据m_already_send_tasks队列也重新放回output_channel:

void net_channel::on_disconnected()
{
	std::vector<msg_task> temp_all_sending_tasks;
	temp_all_sending_tasks.reserve(m_sending_tasks.size() + m_already_send_tasks.size());
	while(!m_already_send_tasks.empty())
	{
		temp_all_sending_tasks.push_back(m_already_send_tasks.front());
		m_already_send_tasks.pop_front();
	}
	while(!m_sending_tasks.empty())
	{
		temp_all_sending_tasks.push_back(m_sending_tasks.front());
		m_sending_tasks.pop_front();
	}
	output_channel->push_front_msg_bulk(temp_all_sending_tasks);
}

同时在network_router处理断线的函数里，将output_channel里的所有数据都重新推送回anchor_resource内部的一个数据队列output_channel中：

bool network_router::disconnect(const net_connection* connection)
{
	auto connection_iter = m_connection_resources.find(connection);
	if(connection_iter == m_connection_resources.end())
	{
		m_logger->warn("disconnect {} fail", connection->get_remote_endpoint_name());

		return false;
	}
	m_inbound_connections.erase(connection->inbound_connection_idx);
	const auto& cur_con_name = connection->get_connection_name();
	if (cur_con_name)
	{
		m_named_connection.erase(*cur_con_name);
		m_logger->info("erase conn {} {}", *cur_con_name, connection->get_remote_endpoint_name());
	}
	auto& cur_connection_resource = connection_iter->second;
	for(auto one_node: cur_connection_resource->anchors)
	{
		m_anchor_collection.disconnect(one_node, connection);
		
	}
	cur_connection_resource->anchors.clear();
	cur_connection_resource->connection_controller->on_disconnected();
	std::array<network_channel_task<std::string, std::string>, 10> temp_task;
	std::vector< network_channel_task<std::string, std::string>> total_tasks;
	std::size_t temp_count;
	while(true)
	{
		temp_count = cur_connection_resource->output_channel->pop_bulk_msg(temp_task.data(), temp_task.size());
		if(temp_count == 0)
		{
			break;
		}
		else
		{
			total_tasks.insert(total_tasks.end(), temp_task.begin(), temp_task.begin() + temp_count);
		}
	}
	on_disconnect_handle_remain_tasks(connection, total_tasks);
	
	m_connection_resources.erase(connection_iter);
	return true;
}
void network_router::on_disconnect_handle_remain_tasks(const net_connection* connection, std::vector< network_channel_task<std::string, std::string>>& remain_tasks)
{
	const auto& cur_con_name = connection->get_connection_name();
	for (std::size_t i = 0; i < remain_tasks.size(); i++)
	{
		auto& cur_task = remain_tasks[remain_tasks.size() - i - 1];
		if (*cur_task.from == *cur_con_name)
		{
			// from is local server  ignore control msg
			continue;
		}
		if(cur_task.dest->empty())
		{
			cur_task.dest = cur_con_name;
		}
		m_anchor_collection.try_push_front(cur_task);
	}
}

这里的anchor_resource的output_channel主要负责在断线的时候缓存住所有要发送的数据，这样就避免了无连接时的数据丢失：

bool anchor_collection::try_push_front(msg_task& task)
{
	auto cur_iter = m_anchor_resources.find(*task.dest);
	if (cur_iter == m_anchor_resources.end())
	{
		return false;
	}
	
	if (cur_iter->second->connection)
	{
		// 在有连接的时候是不能这么做的
		// 以防老连接在断线时还没发出的信息插入到新连接里
		return false;
	}
	cur_iter->second->output_channel.push_front_msg(task);
	return true;
}

同时在basic_stub处理断线的时候，需要使用should_reconnect检查当前连接是否需要重连，如果需要重连，立即开启下一帧的计时器来启动重连:

void basic_stub::on_disconnected(std::shared_ptr<network::net_connection> connection)
{
	m_router->disconnect(connection);

	const auto& cur_connection_name = connection->get_connection_name();
	if (!cur_connection_name || cur_connection_name->empty())
	{
		m_logger->info("empty conn name for {}", connection->get_remote_endpoint_name());
		return;
	}

	if (!should_reconnect(connection))
	{
		return;
	}

	// re connect
	m_logger->info("reconnect to server {} after {} ms", *cur_connection_name, m_timer_check_gap_ms);
	add_timer_with_gap(std::chrono::milliseconds(m_timer_check_gap_ms), [cur_connection_name, this]()
		{
			connect_to_server(*cur_connection_name);
		});
}

上面这些就是MosaicGame网络层为断线重连做的一些底层支持，但是这些代码只是做到了未发送数据和一些已发送数据的缓存。断线重连时，还需要设计额外的逻辑来做逻辑层的接收确认，整个接收确认以及重传的逻辑链路还是比较长的，需要详细的跟踪这期间的流程：

1. 客户端第一次连接到gate_server时，会向gate_server发送一个request_create_session的请求：

void basic_client::on_connect(std::shared_ptr<network::net_connection> connection)
{
	basic_stub::on_connect(connection);

	auto cur_con_name = get_connection_name(connection.get());
	if (cur_con_name && *cur_con_name == m_upstream_server.name)
	{
		m_gate_connection = connection;
		m_logger->info("set gate connection with name {}", *cur_con_name);
		if (!m_main_player)
		{
			request_create_session(connection);
		}
		else
		{
			request_reconnect_session(connection);
		}	
	}
}

void basic_client::request_create_session(std::shared_ptr<network::net_connection> connection)
{
	json message;
	message["cmd"] = "request_create_session";
	json::object_t params;
	message["param"] = params;
	m_router->push_msg(connection.get(), m_local_name_ptr, get_connection_name(connection.get()), std::make_shared<const std::string>(message.dump(4)),  enums::packet_cmd_helper::encode(enums::packet_cmd::server_control, 0));
}

2. gate_server分配一个唯一的session_str作为两者session的唯一标识符，以及一个account_id,一并发送到客户端：

void gate_server::on_request_create_session(std::shared_ptr<network::net_connection> con, std::shared_ptr<const std::string> dest, const json& msg)
{
	if(m_stopped)
	{
		return;
	}
	std::string error_info = std::string();
	std::string cur_session_str;
	std::shared_ptr<network::net_connection> outbound_con;
	do {
		if (m_connection_sessions.find(con->inbound_connection_idx) != m_connection_sessions.end())
		{
			error_info = "already has session";
			break;
		}
		outbound_con = choose_space_server();
		if (!outbound_con)
		{
			error_info = "no game server available";
			break;
		}
		cur_session_str = generate_session_str();
	} while (0);

	json reply_msg, reply_param;
	reply_msg["cmd"] = "reply_create_session";
	reply_param["errcode"] = error_info;
	if (error_info.empty())
	{
		reply_param["account_id"] = on_session_created(con, outbound_con, cur_session_str);
	}
	else
	{
		reply_param["account_id"] = std::string{};
	}
	reply_param["session"] = cur_session_str;

	reply_msg["param"] = reply_param;

	m_router->push_msg(con.get(), m_local_name_ptr, {}, std::make_shared<const std::string>(reply_msg.dump()), enums::packet_cmd_helper::encode(enums::packet_cmd::server_control, 0));


}

3. gate_server以这个session_str作为anchor分配一个anchor_resource并关联当前连接，这样所有服务端发向这个客户端的数据都会通过这个anchor_resource内的队列进行中转， 同时根据当前的session关联的entity_id初始化数据包编号为0：

std::string gate_server::on_session_created(std::shared_ptr<network::net_connection> inbound_con, std::shared_ptr<network::net_connection> outbound_con, const std::string& session_key)
{
	auto cur_inbound_con_idx = inbound_con->inbound_connection_idx;
	session_info cur_session_info;
	cur_session_info.session = session_key;
	cur_session_info.inbound_con = inbound_con;
	cur_session_info.outbound_con = outbound_con;
	cur_session_info.entity_id = generate_account_id(cur_inbound_con_idx);
	cur_session_info.shared_eid = std::make_shared<std::string>(cur_session_info.entity_id);
	m_connection_sessions[cur_inbound_con_idx] = cur_session_info;
	m_session_to_conn_id[session_key] = cur_inbound_con_idx;
	m_eid_to_conn_id[cur_session_info.entity_id] = cur_inbound_con_idx;
	m_router->link_anchor_to_connection(cur_session_info.entity_id, inbound_con.get());
	inbound_con->set_connection_name(cur_session_info.entity_id, m_logger);
	m_logger->info("link_anchor_to_connection eid {} con_id {}", cur_session_info.entity_id, inbound_con->inbound_connection_idx);
	json create_account_info, create_param, init_info;
	create_account_info["cmd"] = "request_create_account";
	create_param["entity_id"] = cur_session_info.entity_id;
	init_info["connection_idx"] = cur_inbound_con_idx;
	create_param["init_info"] = init_info;
	create_account_info["param"] = create_param;
	m_router->push_msg(outbound_con.get(), m_local_name_ptr, outbound_con->get_connection_name(), std::make_shared<std::string>(create_account_info.dump(4)), enums::packet_cmd_helper::encode(enums::packet_cmd::server_control, 0));
	// 新创建的时候 初始化序列号为0
	m_entity_send_last_seq[cur_session_info.entity_id] = 0;
	return cur_session_info.entity_id;
}

4. 当gate_server往客户端发送消息时，会对当前数据包的msg_seq做自增：

// game 发往client
void gate_server::on_call_client(const network::msg_task& one_msg)
{

	m_logger->debug("call client {} with msg {}", *one_msg.dest, *one_msg.data);
	auto temp_iter = m_entity_send_last_seq.find(*one_msg.dest);
	if(temp_iter == m_entity_send_last_seq.end())
	{
		return;
	}
	// 添加唯一有序递增编号
	temp_iter->second++;
	auto cur_msg_seq = temp_iter->second;

	// game发向client 时 直接使用entity id 而不要去查询connection
	// 因为此时可能在断线重连阶段 我们要利用这个按名发送的缓冲机制缓存一下数据
	m_router->push_msg({}, one_msg.dest, one_msg.data, one_msg.cmd, cur_msg_seq);
}

5. 一个客户端连接断线时，net_channel会把所有未发送数据和一些已发送数据有序的保存在anchor_resource的output_channel中，同时保留这个anchor_resource一段时间作为重连时间窗口，这样服务端发往此客户端的消息会暂存到这个anchor_resource内部的消息队列output_channel中，不至于丢失消息：

void gate_server::on_client_disconnected(std::shared_ptr<network::net_connection> connection)
{
	auto cur_con_idx = connection->inbound_connection_idx;
	auto cur_session_iter = m_connection_sessions.find(cur_con_idx);
	if (cur_session_iter == m_connection_sessions.end())
	{
		return;
	}
	cur_session_iter->second.inbound_con.reset();
	m_session_disconnected_ts[cur_con_idx] = std::chrono::system_clock::now();
	
	json notify_client_disconnected_info, param;
	notify_client_disconnected_info["cmd"] = "notify_client_disconnected";
	param["entity_id"] = cur_session_iter->second.entity_id;
	notify_client_disconnected_info["param"] = param;
	m_router->push_msg(cur_session_iter->second.outbound_con.get(), m_local_name_ptr, {}, std::make_shared<std::string>(notify_client_disconnected_info.dump(4)), enums::packet_cmd_helper::encode(enums::packet_cmd::server_control, 0));
}
void gate_server::on_disconnected(std::shared_ptr<network::net_connection> connection)
{
	if (!connection->is_outbound_connection())
	{
		on_client_disconnected(connection);
	}
	else
	{
		on_lose_game(connection);
	}
	json_stub::on_disconnected(connection);
}

服务端对应的account_entity会收到客户端断线的通知，不过这里并不会做特殊的处理，只是打印一下日志：

void account_entity::on_notify_client_disconnected()
{
	m_logger->info("{} on_notify_client_disconnected ", m_base_desc.m_persist_entity_id);
}

6. 这里还记录了一个断线时间戳数据到m_session_disconnected_ts中，gate_server会定期扫描其中掉线时间超过m_lost_client_gap的session，来释放资源，同时通知对应的服务端account_entity客户端掉线的消息：

void gate_server::check_remove_session()
{
	std::vector<std::uint64_t> temp_con_to_remove;
	auto now_ts = std::chrono::system_clock::now();
	for(auto one_pair: m_session_disconnected_ts)
	{
		auto temp_duration = std::chrono::duration_cast<std::chrono::seconds>(now_ts - one_pair.second);
		if(temp_duration.count() > m_lost_client_gap)
		{
			temp_con_to_remove.push_back(one_pair.first);
		}
	}
	for(auto one_con: temp_con_to_remove)
	{
		m_session_disconnected_ts.erase(one_con);
		auto cur_iter = m_connection_sessions.find(one_con);
		if(cur_iter == m_connection_sessions.end())
		{
			continue;
		}
		const auto& cur_session = cur_iter->second.session;
		m_logger->info("remove expired session {} with entity id {}", cur_session, cur_iter->second.entity_id);
		json notify_client_destroy_info, param;
		notify_client_destroy_info["cmd"] = "notify_client_destroy";
		param["entity_id"] = cur_iter->second.entity_id;
		notify_client_destroy_info["param"] = param;
		m_router->push_msg(cur_iter->second.outbound_con.get(), m_local_name_ptr, {}, std::make_shared<std::string>(notify_client_destroy_info.dump(4)), enums::packet_cmd_helper::encode(enums::packet_cmd::server_control, 0));
		for(auto& one_group: m_clients_for_group)
		{
			one_group.second.erase(std::string_view(cur_iter->second.entity_id));
		}
		m_session_to_conn_id.erase(cur_session);
		m_eid_to_conn_id.erase(cur_iter->second.entity_id);
		m_router->remove_anchor(cur_iter->second.entity_id);
		
		m_connection_sessions.erase(cur_iter);

	}
	m_session_remove_timer = add_timer_with_gap(std::chrono::seconds(1), [this]() 
	{
		this->check_remove_session();
	});
}

7. 客户端发现自己断线后，如果发现当前已经登陆完成并创建好了玩家角色，则会重新发起到原始上游gate_server的连接，同时记录当前已经接收到的最大消息包序列号到m_reconnect_msg_read_seq，如果没有创建玩家角色，则彻底断线：

bool basic_client::should_reconnect(std::shared_ptr<network::net_connection> connection)
{
	if (!basic_stub::should_reconnect(connection))
	{
		return false;
	}
	if(connection == m_gate_connection)
	{
		// 如果是gate连接 则只有在角色创建之后走重连逻辑 否则走重新登录
		// 如果这个read_seq 为0 代表业务消息包还暂未发送 或者最新的包是控制消息包
		return m_main_player != nullptr && m_router->get_connection_resource(m_gate_connection)->connection_controller->get_packet_read_seq() != 0;
	}
	else
	{
		return true;
	}
}
void basic_stub::on_disconnected(std::shared_ptr<network::net_connection> connection)
{
	m_router->disconnect(connection);

	const auto& cur_connection_name = connection->get_connection_name();
	if (!cur_connection_name || cur_connection_name->empty())
	{
		m_logger->info("empty conn name for {}", connection->get_remote_endpoint_name());
		return;
	}

	if (!should_reconnect(connection))
	{
		return;
	}

	// re connect
	m_logger->info("reconnect to server {} after {} ms", *cur_connection_name, m_timer_check_gap_ms);
	add_timer_with_gap(std::chrono::milliseconds(m_timer_check_gap_ms), [cur_connection_name, this]()
		{
			connect_to_server(*cur_connection_name);
		});
}
void basic_client::on_disconnected(std::shared_ptr<network::net_connection> connection)
{
	bool is_lose_server = connection == m_gate_connection;
	json_stub::on_disconnected(connection);
	if (is_lose_server)
	{
		if (m_main_player)
		{
			m_main_player->on_lose_server();
		}
		else
		{
			if (m_main_account)
			{
				m_main_account->on_lose_server();
				entity::entity_manager::instance().destroy_entity(m_main_account);
				m_main_account = nullptr;
			}
		}
		m_gate_connection = nullptr;
	}
}

8. 当连接到gate_server成功后，会触发on_connect回调，内部判断角色已经创建的情况下会发送一个重连消息包，附上之前商定好的session_str以及本地接收到的最大包序列号msg_read_seq:

void basic_client::on_connect(std::shared_ptr<network::net_connection> connection)
{
	basic_stub::on_connect(connection);

	auto cur_con_name = get_connection_name(connection.get());
	if (cur_con_name && *cur_con_name == m_upstream_server.name)
	{
		m_gate_connection = connection;
		m_logger->info("set gate connection with name {}", *cur_con_name);
		if (!m_main_player)
		{
			request_create_session(connection);
		}
		else
		{
			request_reconnect_session(connection);
		}	
	}
}

void basic_client::request_reconnect_session(std::shared_ptr<network::net_connection> connection)
{
	json message;
	message["cmd"] = "request_reconnect_session";
	json::object_t params;
	params["pre_session"] = m_session_key;
	params["msg_read_seq"] = m_router->get_connection_resource(m_gate_connection)->connection_controller->get_packet_read_seq();
	message["param"] = params;
	m_router->push_msg(connection.get(), m_local_name_ptr, get_connection_name(connection.get()), std::make_shared<const std::string>(message.dump(4)), enums::packet_cmd_helper::encode(enums::packet_cmd::server_control, 0));
}

9. gate_server收到一个连接的重连消息包之后，检查session_str是否仍然有效，如果有效则将对应的anchor_resource绑定到新的连接，否则通知客户端重连失败，退回到等待登录状态:

void gate_server::on_request_reconnect_session(std::shared_ptr<network::net_connection> con, std::shared_ptr<const std::string> dest, const json& msg)
{
	if(m_stopped)
	{
		return;
	}
	std::string pre_session;
	std::uint64_t last_read_msg_seq = 0;
	std::uint64_t pre_connection_idx = 0;
	std::string error_info = std::string();

	try
	{
		msg.at("pre_session").get_to(pre_session);
		msg.at("msg_read_seq").get_to(last_read_msg_seq);
	}
	catch (std::exception& e)
	{
		m_logger->error("on_request_reconnect_session fail to parse {} error {}", msg.dump(), e.what());
		error_info = "invalid msg format";
	}

	if (error_info.empty())
	{
		do
		{
			if (m_connection_sessions.find(con->inbound_connection_idx) != m_connection_sessions.end())
			{
				error_info = "already has session";
				break;
			}
			auto cur_con_id_iter = m_session_to_conn_id.find(pre_session);
			if(cur_con_id_iter == m_session_to_conn_id.end())
			{
				error_info = "invalid session key";
				break;
			}
			pre_connection_idx = cur_con_id_iter->second;
			auto cur_session_iter = m_connection_sessions.find(pre_connection_idx);
			if (cur_session_iter == m_connection_sessions.end())
			{
				error_info = "invalid pre_connection";
				break;
			}
			if (cur_session_iter->second.session != pre_session)
			{
				error_info = "session not match";
				break;
			}
			if (cur_session_iter->second.inbound_con)
			{
				error_info = "session still online";
				break;
			}
			if(!m_router->remove_readed_msgs(cur_session_iter->second.entity_id, last_read_msg_seq))
			{
				error_info = "invalid last_read_msg_seq";
				break;
			}

		} while (0);
	}
	
	json reply_msg, reply_param;
	reply_msg["cmd"] = "reply_reconnect_session";
	reply_param["errcode"] = error_info;
	reply_msg["param"] = reply_param;
	m_router->push_msg(con.get(), m_local_name_ptr, {}, std::make_shared<const std::string>(reply_msg.dump()), enums::packet_cmd_helper::encode(enums::packet_cmd::server_control, 0));
	if (error_info.empty())
	{
		on_session_reconnected(pre_connection_idx, con, last_read_msg_seq);
	}
	
	return;
}

10. 这里的remove_readed_msgs会将anchor_resource的输出队列中所有packet_seq小于等于last_read_msg_seq的数据都删除，因为这些数据已经被客户端确认过了。如果队列中剩余消息包的最小编号大于last_read_msg_seq+1，则代表已确认数据与剩余数据之间有空窗，出现消息丢失，导致无法重连。

bool anchor_collection::remove_readed_msgs(const std::string& anchor_name, std::uint64_t last_read_msg_seq)
{
	auto cur_iter = m_anchor_resources.find(anchor_name);
	anchor_resource* cur_resource = nullptr;
	if (cur_iter != m_anchor_resources.end())
	{
		cur_resource = cur_iter->second.get();
	}
	if (!cur_resource)
	{
		return false;
	}
	if(cur_resource->get_connection())
	{
		return false;
	}
	std::array<network_channel_task<std::string, std::string>, 10> temp_tasks;
	std::size_t cur_count = 0;
	std::size_t min_msg_seq_in_queue = 0; // 记录队列里的最小数据编号
	while (true)
	{
		cur_count = cur_resource->output_channel.pop_bulk_msg(temp_tasks.data(), temp_tasks.size());
		if (cur_count == 0)
		{
			break;
		}
		if(min_msg_seq_in_queue==0)
		{
			min_msg_seq_in_queue = temp_tasks[0].msg_seq;
		}
		for(std::uint32_t i = 0; i< cur_count;i++)
		{
			
			if(temp_tasks[i].msg_seq >last_read_msg_seq)
			{
				std::vector<network_channel_task<std::string, std::string>> remain_tasks(temp_tasks.data() + i, temp_tasks.data() + cur_count);
				cur_resource->output_channel.push_front_msg_bulk(remain_tasks);
				break;
			}
		}
	}
	// 如果出现最小包编号大于已读编号加1 则代表断线重连会出现消息丢失 此时返回false
	return min_msg_seq_in_queue<=last_read_msg_seq+1; 
}

11. 如果传递过来的session_str是一个有效的已经断线且还没有完全过期的session，同时传递过来的last_read_msg_seq+1大于等于发送缓冲区里最小的数据编号，则将这个session之前绑定的客户端连接替换为当前的新连接：

void gate_server::on_session_reconnected(std::uint64_t pre_con_idx, std::shared_ptr< network::net_connection> new_con)
{
	session_info cur_session_info;
	auto pre_session_iter = m_connection_sessions.find(pre_con_idx);
	cur_session_info.session = pre_session_iter->second.session;
	cur_session_info.inbound_con = new_con;
	cur_session_info.outbound_con = pre_session_iter->second.outbound_con;
	cur_session_info.entity_id = pre_session_iter->second.entity_id; // 这里维持原来的entity_id 因为game上使用最开始的entity_id创建的account
	cur_session_info.shared_eid = std::make_shared<std::string>(cur_session_info.entity_id);
	m_connection_sessions.erase(pre_session_iter);
	m_connection_sessions[new_con->inbound_connection_idx] = cur_session_info;
	m_session_to_conn_id[cur_session_info.session] = new_con->inbound_connection_idx;
	m_eid_to_conn_id[cur_session_info.entity_id] = new_con->inbound_connection_idx;
	new_con->set_connection_name(cur_session_info.entity_id, m_logger);
	m_router->link_anchor_to_connection(cur_session_info.entity_id, new_con.get());

	m_session_disconnected_ts.erase(pre_con_idx);
	json notify_client_reconnected_info, param;
	notify_client_reconnected_info["cmd"] = "notify_client_reconnected";
	param["entity_id"] = cur_session_info.entity_id;
	notify_client_reconnected_info["param"] = param;
	m_router->push_msg(cur_session_info.outbound_con.get(), m_local_name_ptr, {}, std::make_shared<std::string>(notify_client_reconnected_info.dump(4)), enums::packet_cmd_helper::encode(enums::packet_cmd::server_control, 0));
}

如果断线重连成功，account_entity会收到notify_client_reconnected通知，这里同样也只是日志记录一下，因此完整的一次断线重连对于player_entity来说是完全无感知的：

void account_entity::on_notify_client_reconnected()
{
	m_logger->info("{} on_notify_client_reconnected ", m_base_desc.m_persist_entity_id);
}

如果在gate_server指定的时间窗口内都没有断线重连回来，则对应的account_entity会收到一个notify_client_destroy的消息，这里会开启一个计时器，如果这个计时器超时前没有新的客户端登录这个账号，则开启自动下线流程：

void account_entity::on_notify_client_destroy()
{
	m_destroy_client_timer = add_timer_with_gap(std::chrono::seconds(m_auto_logout_second_when_client_destroy), [this]()
	{
		request_logout_account(utility::rpc_msg());
	});
}

迁移保序

在我们目前的服务器架构中， space_server是可以有多个实例，如果一个actor_entity有迁移能力，那么这个actor_entity就可能会在不同的进程中进行迁移。由于迁移前后同一个actor_entity的通信地址会发生变化，所以如果直接使用actor_entity的当前地址作为投递地址的话很可能出现迁移后消息丢失的情况。为了避免出现迁移引发的地址失效问题，对于可以迁移的actor_entity，我们会在创建这个actor_entity的同时，创建一个不参与迁移的relay_entity。这个relay_entity的作用就是作为这个actor_entity的消息中转站来使用，实现方式是将这个不会迁移的relay_entity的call_anchor设置到这个actor_entity上。 player_entity创建时会直接在初始数据里带上call_anchor参数，而其他普通的actor_entity则是在创建之后手动调用set_call_anchor来修改内部的m_call_anchor:

player_entity* account_entity::create_player_entity(const std::string& player_id, const json& player_doc)
{
	std::string cur_err;
	auto cur_relay_entity_id = std::to_string(get_server()->gen_unique_uint64());
	json::object_t relay_init_info;
	relay_init_info["dest_eid"] = player_id;
	relay_init_info["dest_game"] = get_server()->local_stub_info().name;
	auto cur_relay_entity = get_server()->create_entity("relay_entity", cur_relay_entity_id, gen_online_entity_id(),relay_init_info, cur_err);
	if(!cur_relay_entity)
	{
		m_logger->error("fail to create relay_entity");
		return nullptr;
	}
	m_relay_entity = dynamic_cast<relay_entity*>(cur_relay_entity);
	m_relay_entity->setup_client_info(m_gate_id, get_call_proxy());
	json::object_t player_init_info;
	// 省略一些代码
	player_init_info["call_proxy"] = *cur_relay_entity->get_call_proxy();
	auto cur_entity = get_server()->create_entity("player_entity", player_id, gen_online_entity_id(),player_init_info, cur_err);
	// 省略后续代码
}

actor_entity* space_entity::create_entity(const std::string& entity_type, const std::string& entity_id, json::object_t& init_info, const json::object_t& enter_info, std::uint64_t online_entity_id)
{
	utility::entity_load_stat_recorder temp_recorder(entity_load_stat());
	if(online_entity_id == 0)
	{
		online_entity_id = gen_online_entity_id();
	}

	std::string create_entity_error;
	auto cur_entity = get_server()->create_entity(entity_type, entity_id, online_entity_id, init_info, create_entity_error);
	if(!cur_entity)
	{
		m_logger->error("fail to create_entity type {} id {} with error {}", entity_type, entity_id, create_entity_error);
		return nullptr;
	}
	auto cur_actor_entity = dynamic_cast<actor_entity*>(cur_entity);
	if(!cur_actor_entity)
	{
		m_logger->error("fail to create actor_entity with entity_type {}", cur_entity->type_name());
		get_server()->destroy_entity(cur_entity);
		return nullptr;
	}
	if(!cur_actor_entity->is_global_actor() && is_cell_space() && !cur_actor_entity->is_ghost())
	{
		// 非全局actor 都需要建立一个relay entity
		json::object_t relay_init_info;
		relay_init_info["dest_game"] = get_server()->local_stub_info().name;
		relay_init_info["dest_eid"] = entity_id;
		auto cur_relay_entity_id = get_server()->gen_unique_str();
		auto cur_relay_entity = get_server()->create_entity("relay_entity", cur_relay_entity_id, gen_online_entity_id(),  relay_init_info, create_entity_error);
		if(!cur_relay_entity)
		{
			m_logger->error("fail to create relay_entity id {} with error {}", entity_id, create_entity_error);
			get_server()->destroy_entity(cur_entity);
			return nullptr;
		}
		
		m_relay_entities[entity_id] = cur_relay_entity;
		cur_actor_entity->set_call_proxy(*cur_relay_entity->get_call_proxy());
		m_logger->info("create relay entity {} for entity  {}", cur_relay_entity_id, entity_id);
	}
	// 省略一些代码
}

当其他模块需要往这个actor_entity发送一个远程消息时，使用的其实就是relay_entity的call_anchor，此时relay_entity处理rpc的时候如果发现自身没有相关rpc的定义，就会将这个数据转发到相应的actor_entity的最新通信地址上：

utility::rpc_msg::call_result relay_entity::on_rpc_msg(const utility::rpc_msg& msg)
{
	auto temp_result = rpc_owner_on_rpc(msg);
	if(temp_result == utility::rpc_msg::call_result::dest_not_found)
	{
		forward_to_actor(msg);
		return utility::rpc_msg::call_result::suc;
	}
	else
	{
		return temp_result;
	}
}

void account_entity::call_player(const utility::rpc_msg& msg)
{
	if(!m_relay_entity)
	{
		return;
	}
	
	m_relay_entity->forward_to_actor(msg);
}

为了在relay_entity上获得对应actor_entity的最新通信地址，需要actor_entity在迁移之前先获得当前relay_entity的允许，此时relay_entity将目标地址设置为空，在目标地址为空的时候，forward_to_actor的消息会先缓存起来，避免迁移中间的消息丢失：

void relay_entity::request_migrate_begin(const utility::rpc_msg& msg, const std::string& game_id, const std::string& space_id, const std::string& union_space_id, const json::object_t& enter_info)
{
	if(!m_dest_actor)
	{
		m_logger->error("request_migrate_begin while dest_anchor empty dest_game {} dest_eid {}", m_dest_game, m_dest_eid);
		return;
	}
	utility::rpc_msg reply_msg;
	reply_msg.cmd = "reply_migrate_begin";
	reply_msg.args.push_back(game_id);
	reply_msg.args.push_back(space_id);
	reply_msg.args.push_back(union_space_id);
	reply_msg.args.push_back(enter_info);
	call_server(m_dest_actor, reply_msg);
	m_dest_actor.reset();
	m_dest_game = game_id;
	
}

void relay_entity::forward_to_actor(const network::msg_task& cur_msg_task)
{
	if(!m_dest_actor)
	{
		m_cached_msgs.push_back(cur_msg_task);
	}
	else
	{
		auto cur_cmd_detail = enums::packet_cmd_helper::decode(cur_msg_task.cmd);
		call_server(m_dest_actor, cur_msg_task.data, cur_cmd_detail.first, cur_cmd_detail.second);
	}
}

actor_entity只有在接收到reply_migrate_begin之后才会开始真正的迁移，当收到这个消息时，之前通过relay_entity发送到当前actor_entity的消息肯定已经全都收到了，因为底层对应的是同一个物理连接。等到actor_entity迁移完成之后，会通过notify_migrate_finish将最新地址发送过来， 此时再将迁移期间缓存的数据按序发出，并清空这个m_cached_msgs数组：

void relay_entity::notify_migrate_finish(const utility::rpc_msg& msg, const std::string& game_id)
{
	if(m_dest_game != game_id)
	{
		m_logger->error("notify_migrate_finish while  game not match  empty dest_game {} dest_eid {} new_game_id {}", m_dest_game, m_dest_eid, game_id);
		return;
	}
	m_dest_actor = std::make_shared<std::string>(utility::rpc_anchor::concat(m_dest_game, m_dest_eid));
	
	for(const auto& one_msg: m_cached_msgs)
	{
		auto cur_cmd_detail = enums::packet_cmd_helper::decode(one_msg.cmd);
		call_server(m_dest_actor, one_msg.data, cur_cmd_detail.first, cur_cmd_detail.second);
	}
	
	m_cached_msgs.clear();
}

当m_dest_actor被设置到值之后，后续的消息发送就直接走call_server接口，不再缓存。

不过relay_entity也有其局限性，他只能保证一个不会迁移的消息发送者发送到当前对应的actor_entity的所有数据按照发出顺序投递。如果消息发送者A在迁移前给relay_entity(B)发出了消息M，在迁移后给relay_entity(B)发出了消息N，那relay_entity(B)只能保证M,N都能投递到actor_entity(B)，不能保证M在N之前到达。因为relay_entity(B)上M与N的接收顺序是不确定的，同时relay_entity(B)只能按照消息到达自身的顺序去转发到actor_entity(B)。如果实在是要求所有由A发送到B的消息必须按照发送顺序来接收，可以按照这个方案来操作：

1. 给A也创建一个relay_entity，
2. A发送给B的数据先发送到relay_entity(A)，此时能够保证relay_entity(A)接受数据的顺序等于A发出数据的顺序
3. 然后relay_entity(A)将数据依次转发到relay_entity(B)，由于两个relay_entity都是不会迁移的，因此这些消息使用的是同一个物理连接，可以保证接受顺序等于发送顺序
4. relay_entity(B)将所有接收到的数据按照顺序发送到actor_entity(B)，这个actor_entity(B)接收顺序也是可以保证的

所以这种强一致性的网络发送方案可以通过两层relay_entity来做到，代价就是消息的延迟增加了两个relay_entity的中转。

迁移时除了往actor_entity投递的消息可能发生乱序和丢失之外，player_entity往客户端发送的消息也可能出现乱序。因为gate_server是不动的，而player_entity则是不断的在进程间迁移的。由于不同进程之间的网络延迟不同，如果player_entity发送给客户端的数据是直接通过gate_server的通信地址直接投递的话，其接收顺序是不能保证的。如果客户端接收到的数据顺序是错乱的，那么很可能会导致各种逻辑问题，典型症状就是同一个属性修改多次的情况下，客户端最后的值与服务端最后的值不一样。

要解决这个顺序问题，就需要跟前述的actor_entity之间通信保序方案一样，在relay_entity上做转发操作，因为当前actor_entity的迁移会在对应的relay_entity之间做一次rpc同步。此时player_entity发往客户端的数据不再直接发送到gate_server，而是先发送到relay_entity，然后再由relay_entity发送到gate_server:

void relay_entity::forward_to_client(const network::msg_task& cur_msg)
{
	if(m_gate_id == 0)
	{
		m_logger->error("fail to forward to client due to gate_id is 0");
		return;
	}
	auto cur_cmd_detail = enums::packet_cmd_helper::decode(cur_msg.cmd).second;
	auto cur_entity_packet_detail = enums::client_entity_packet_helper::decode(cur_cmd_detail);
	if(cur_entity_packet_detail.second != m_gate_version)
	{
		m_logger->error("fail to forward to client due to gate_version not match gate_version {} msg_gate_version {}", m_gate_version, cur_entity_packet_detail.second);
		return;
	}
	get_server()->call_gate(m_gate_id, m_account_anchor, cur_msg.data,cur_entity_packet_detail.first);
}

void player_entity::call_client(enums::entity_packet entity_packet_cmd, std::shared_ptr<const std::string> data)
{
	if(!has_client())
	{
		return;
	}
	if(entity_packet_cmd == enums::entity_packet::json_rpc || entity_packet_cmd == enums::entity_packet::sync_prop)
	{
		m_logger->debug("call_client  cmd {} data {}", std::uint8_t(entity_packet_cmd), *data);
	}
	else
	{
		m_logger->debug("call_client cmd {} sz {}", std::uint8_t(entity_packet_cmd), data->size());
	}
	if(is_ghost())
	{
		m_logger->warn("ghost player_entity call_client ignored");
		return;
	}
	if(b_is_migrating)
	{
		m_to_client_msg_when_migrating.emplace_back(enums::client_entity_packet_helper::encode(entity_packet_cmd, m_gate_version), data);
	}
	else
	{
		call_relay_anchor(enums::packet_cmd::game_to_client, enums::client_entity_packet_helper::encode(entity_packet_cmd, m_gate_version), data);
	}
}

由于actor_entity向relay_entity执行rpc同步是一个异步操作，因此在这个操作期间发往客户端的数据不能直接发送到relay_entity，而是先缓存到自身的一个临时队列m_to_client_msg_when_migrating中。真正开始迁移打包数据的时候会将这个队列一起打包：

void player_entity::encode_migrate_out_data(json::object_t& migrate_info, bool enter_new_space)
{
	actor_entity::encode_migrate_out_data(migrate_info, enter_new_space);
	migrate_info["prop"] = m_prop_data.encode();
	migrate_info["gate_version"] = m_gate_version;
	std::vector<json> cached_client_msgs;
	for(auto& [cmd, msg]: m_to_client_msg_when_migrating)
	{
		json one_msg = serialize::encode_multi(cmd, *msg);
		cached_client_msgs.push_back(std::move(one_msg));
	}
	migrate_info["cached_to_client_msgs"] = cached_client_msgs;
	m_to_client_msg_when_migrating.clear();
}

当迁移完毕之后，会先通知relay_entity迁移完成，然后再将这个缓存的客户端数据依次发出：

void player_entity::player_event_listener(const utility::enum_type_value_pair& ev_cat, const json::object_t& detail)
{
	if(ev_cat == utility::enum_type_value_pair(enums::migrate_event::migrate_in_after_component_decode))
	{
		if(is_ghost())
		{
			return;
		}
		auto cur_gate_version_iter = detail.find("gate_version");
		if(cur_gate_version_iter != detail.end())
		{
			m_gate_version = cur_gate_version_iter->second.get<std::uint8_t>();
		}
		else
		{
			m_logger->error("migrate_in_after_component_decode missing gate_version set to 0");
			m_gate_version = 0;
		}
		if(!has_client())
		{
			return;
		}
		auto cached_msgs_iter = detail.find("cached_to_client_msgs");
		if(cached_msgs_iter == detail.end())
		{
			return;
		}
		auto& cached_msgs = cached_msgs_iter->second;
		for(const auto& one_cached_msg: cached_msgs)
		{
			std::uint8_t cmd;
			std::string msg;
			serialize::decode_multi(one_cached_msg, cmd, msg);
			call_relay_anchor(enums::packet_cmd::game_to_client, cmd, std::make_shared<const std::string>(std::move(msg)));
		}
	}
}

顶号保护

我们目前的设计里，当多个客户端登录同一个账号时，前面的客户端会被后面的客户端顶掉，这个过程也叫顶号。在顶号发生时，player_entity的gate将会被更新，会先被通知客户端销毁，

void player_entity::notify_player_client_destroyed(const utility::rpc_msg& msg)
{
	m_logger->warn("player {} notify_player_client_replaced", entity_id());
	m_gate_version = 0;
}

然后等新的gate绑定完成时,再被同步最新的gate信息。此时player_entity会将当前的最新数据完整的打包下去，通知客户端来同步当前的所有状态：

void player_entity::notify_player_client_replaced(const utility::rpc_msg& msg, const std::string& new_gate_name, std::uint8_t new_gate_version)
{
	m_logger->warn("player client replaced by new gate {} version {}", new_gate_name, new_gate_version);
	m_gate_version = new_gate_version;
	if(new_gate_name.empty())
	{
		return;
	}

	auto sync_info = encode_with_flag(std::uint32_t(enums::encode_flags::self_client));
	utility::rpc_msg full_sync_msg;
	full_sync_msg.cmd = "create_player";
	full_sync_msg.args.push_back(entity_id());
	full_sync_msg.args.push_back(std::move(sync_info));
	call_client(full_sync_msg);
	m_login_dispatcher.dispatch(true);

}

但是由于在顶号期间，player_entity只是被动的接收这两个rpc，同时player_entity通过relay_entity往客户端发送数据是主动的。所以可能会出现一些本来应该发往老客户端的数据被发到新客户端的情况：

1. 时刻1 player_entity往relay_entity发送一条往客户端Client(A)的消息M
2. 时刻2 player_entity接收到relay_entity转发过来的notify_player_client_destroyed消息，此时relay_entity上的gate信息被清空
3. 时刻3 player_entity接收到relay_entity转发过来的notify_player_client_replaced消息，此时relay_entity上的gate信息被设置为最新的Client(B)
4. 时刻4 消息M在relay_entity上被接收，并通过forward_to_client转发到了Client(B)

为了修正这种错误的客户端数据发送，我们可以在player_entity发送客户端数据时都带上当前的gate信息。当relay_entity接收到这个数据的时候，会将消息里的gate信息与当前relay_entity的gate信息进行比对，如果不相等则直接丢弃。

由于顶号是一个低频的操作，为所有给客户端发送的数据都带上一个当前gate的信息会导致数据包被重新打包，比较浪费CPU。因此当前在mosaic_game中，只携带了4bit的gate_version信息，这样这个4bit与entity_packet可以拼接成一个uint8，可以避免数据的重新打包：

static_assert(int(enums::entity_packet::max) <= 16, "entity_packet max exceed 16, cannot fit in 4 bits because gate_version also need 4 bits");

// 将entity_packet与gate_version结合编码解码 用一个字节表示
// 这样就可以方便的区分数据里的gate版本 避免顶号之后还能收到发往老客户端的数据
struct client_entity_packet_helper
{
	static std::uint8_t encode(entity_packet in_packet_cmd, std::uint8_t in_gate_version)
	{
		std::uint8_t result = std::uint8_t(in_packet_cmd);
		result <<= 4;
		result += in_gate_version;
		return result;
	}

	static std::pair<entity_packet, std::uint8_t> decode(std::uint8_t in_combine_cmd)
	{
		std::pair<entity_packet, std::uint8_t> result;
		result.second = in_combine_cmd % 256;
		result.first = entity_packet(in_combine_cmd / 256);
		return result;
	}
};

void player_entity::call_client(enums::entity_packet entity_packet_cmd, std::shared_ptr<const std::string> data)
{
	// 省略很多之前已经介绍的代码
	call_relay_anchor(enums::packet_cmd::game_to_client, enums::client_entity_packet_helper::encode(entity_packet_cmd, m_gate_version), data);
}

当relay_entity在接收到这些向客户端转发的数据时，会将这里面4bit的gate_version拿出来进行比对,不匹配则忽略：

void relay_entity::forward_to_client(const network::msg_task& cur_msg)
{
	if(m_gate_id == 0)
	{
		m_logger->error("fail to forward to client due to gate_id is 0");
		return;
	}
	auto cur_cmd_detail = enums::packet_cmd_helper::decode(cur_msg.cmd).second;
	auto cur_entity_packet_detail = enums::client_entity_packet_helper::decode(cur_cmd_detail);
	if(cur_entity_packet_detail.second != m_gate_version)
	{
		m_logger->error("fail to forward to client due to gate_version not match gate_version {} msg_gate_version {}", m_gate_version, cur_entity_packet_detail.second);
		return;
	}
	get_server()->call_gate(m_gate_id, m_account_anchor, cur_msg.data,cur_entity_packet_detail.first);
}

所以目前只要能够正确的在relay_entity与player_entity之间同步好gate_version，就可以避免老客户端的数据被发送到新客户端。此时只需要在account_entity::set_gate绑定新gate的时候将gate_version进行自增，并发送到player_entity即可：

std::uint8_t relay_entity::gate_version() const
{
	return m_gate_version;
}

void relay_entity::setup_client_info(std::uint64_t in_gate_id, std::shared_ptr<const std::string> in_account_anchor)
{
	m_gate_id = in_gate_id;
	m_account_anchor = std::move(in_account_anchor);
	if(m_gate_id != 0)
	{
		++m_gate_version;
		if(m_gate_version >= 16)
		{
			m_gate_version = 1;
		}
	}
}

void account_entity::set_gate(const std::string& gate_name, std::uint64_t gate_id, bool during_replace)
{
	m_gate_name = gate_name;
	m_gate_id = gate_id;
	m_relay_entity->setup_client_info(m_gate_id, get_call_proxy());
	if(gate_id != 0)
	{
		m_logger->info("{} notify_rebind_gate_client_finish with new_gate {}", m_base_desc.m_persist_entity_id, gate_name);
		cancel_timer(m_destroy_client_timer);
		m_destroy_client_timer.reset();
		if(during_replace)
		{
			// 如果当前正在顶号过程中 通知客户端顶号成功
			utility::rpc_msg replace_info;
			replace_info.cmd = "reply_replace_account";
			replace_info.args.push_back(is_player_online());
			call_client(replace_info);
		}
		if(!is_player_online())
		{
			m_statem.change_to("show_players");
		}
		else
		{
			// 触发重新同步数据
			utility::rpc_msg account_replace_msg;
			account_replace_msg.cmd = "notify_player_client_replaced";
			account_replace_msg.args.push_back(m_gate_name);
			account_replace_msg.args.push_back(m_relay_entity->gate_version());
			call_player(account_replace_msg);
		}
	}
	// 省略后续代码
}

当account_entity创建player_entity时，会做第一次的gate同步：

player_entity* account_entity::create_player_entity(const std::string& player_id, const json& player_doc)
{
	std::string cur_err;
	auto cur_relay_entity_id = std::to_string(get_server()->gen_unique_uint64());
	json::object_t relay_init_info;
	relay_init_info["dest_eid"] = player_id;
	relay_init_info["dest_game"] = get_server()->local_stub_info().name;
	auto cur_relay_entity = get_server()->create_entity("relay_entity", cur_relay_entity_id, gen_online_entity_id(),relay_init_info, cur_err);
	if(!cur_relay_entity)
	{
		m_logger->error("fail to create relay_entity");
		return nullptr;
	}
	m_relay_entity = dynamic_cast<relay_entity*>(cur_relay_entity);
	m_relay_entity->setup_client_info(m_gate_id, get_call_proxy());
	json::object_t player_init_info;
	// 省略很多代码
	auto cur_entity = get_server()->create_entity("player_entity", player_id, gen_online_entity_id(),player_init_info, cur_err);
	if(!cur_entity)
	{
		m_logger->error("fail to create player {} error is {} doc is {}", player_id, cur_err, player_doc.dump());
		request_logout_account(utility::rpc_msg());
		return nullptr;
	}
	
	m_player_id = player_id;
	auto cur_player = dynamic_cast<player_entity*>(cur_entity);
	cur_player->set_gate_version_when_create(m_relay_entity->gate_version());
	// 省略很多代码
}

BigWorld 的网络框架

通信地址 Mailbox

在常规的网络通信中，通信地址一般都是以(ip,port)二元组的形式来构成的。如果对于指定的(ip,port)上有多个可以通信的entity/service实例的话，这个二元组就可以扩充为三元组(ip,port,entity_id/service_id)。在mosaic_game中的anchor就是这样的三元组的实现，不过anchor里用server_name来替代(ip,port)二元组，所以是由(server_name, entity_id/service_id)这样的二元组组成的。在Bigworld里也有一个专门的通信地址类型Address, 里面封装了(ip,port)二元组，然后还加上一个salt，有时候这个salt区分不同时间启动的占用同一个(ip,port)的通信地址,有时候这个字段又可以用来区分不同的Entity类型:

class Address
{
public:
	/// @name Construction/Destruction
	// @{
	Address();
	Address( uint32 ipArg, uint16 portArg );
	// @}

	uint32	ip;		///< IP address.
	uint16	port;	///< The port.
	uint16	salt;	///< Different each time.

	int writeToString( char * str, int length ) const;

	// TODO: Remove this operator
	operator char*() const	{ return this->c_str(); }
	char * c_str() const;
	const char * ipAsString() const;

	bool isNone() const			{ return this->ip == 0; }

	static Watcher & watcher();

	static const Address NONE;

private:
	/// Temporary storage used for converting the address to a string.  At
	/// present we support having two string representations at once.
	static const int MAX_STRLEN = 32;
	static char s_stringBuf[ 2 ][ MAX_STRLEN ];
	static int s_currStringBuf;
	static char * nextStringBuf();
};

然后对于指定进程上的Entity的通信地址，使用专门的EntityMailBoxRef来代表，其实就是在Address的基础上加上了EntityId:

typedef int32 EntityID;
/**
 *	This structure is a packed version of a mailbox for an entity
 */
class EntityMailBoxRef
{
public:
	EntityID			id;
	Mercury::Address	addr;

	enum Component
	{
		CELL = 0,
		BASE = 1,
		CLIENT = 2,
		BASE_VIA_CELL = 3,
		CLIENT_VIA_CELL = 4,
		CELL_VIA_BASE = 5,
		CLIENT_VIA_BASE = 6,
		SERVICE = 7
	};

	EntityMailBoxRef():
		id( 0 ),
		addr( Mercury::Address::NONE )
	{}
	
	bool hasAddress() const 		{ return addr != Mercury::Address::NONE; }

	Component component() const		{ return (Component)(addr.salt >> 13); }
	void component( Component c )	{ addr.salt = type() | (uint16(c) << 13); }

	EntityTypeID type() const		{ return addr.salt & 0x1FFF; }
	void type( EntityTypeID t )		{ addr.salt = (addr.salt & 0xE000) | t; }

	void init() { id = 0; addr.ip = 0; addr.port = 0; addr.salt = 0; }
	void init( EntityID i, const Mercury::Address & a,
		Component c, EntityTypeID t )
	{ id = i; addr = a; addr.salt = (uint16(c) << 13) | t; }

	static const char * componentAsStr( Component component );

	const char * componentName() const
	{
		return componentAsStr( this->component() );
	}
};

这里的枚举类型Component定义了一个EntityMailBoxRef地址的具体类型，目前有8种，复用addr.salt的高3位来表示:

1. CELL，一个在CellApp上的Entity地址，也是最直接的Entity地址
2. BASE，一个在BaseApp上的Base对象的地址，这个Base对象负责控制对应的RealEntity
3. CLIENT，一个客户端地址
4. BASE_VIA_CELL，一个中转用的CELL地址，向这个CELL地址投递的消息在被Entity接受之后，会自动的转发到对应的Base对象上
5. CLIENT_VIA_CELL，一个中转用的CELL地址，向这个CELL地址投递的消息在被Entity接受之后，会自动的转发到对应的客户端对象上
6. CELL_VIA_BASE， 一个中转用的Base地址，向这个BASE地址投递的消息在被Base接收之后，会自动的转发到对应的RealEntity上
7. CLIENT_VIA_BASE，一个中转用的Base地址，向这个BASE地址投递的消息在被Base接收之后，会自动的转发到对应的CLIENT上
8. SERVICE，一个服务地址

EntityMailBoxRef这个类型仅仅是作为通信地址使用的，过于底层。业务层里用来发送消息的基本都是PyEntityMailBox这个类型，因为这个类型除了存储了具体的通信地址之外，还负责对所有发送到这个地址的数据做一些封装转换的工作:

/**
 *	This class is used to represent a destination of an entity that messages
 *	can be sent to.
 *
 *	Its virtual methods are implemented differently on each component.
 */
class PyEntityMailBox: public PyObjectPlus
{
	Py_Header( PyEntityMailBox, PyObjectPlus )

public:
	PyEntityMailBox( PyTypeObject * pType = &PyEntityMailBox::s_type_ );
	virtual ~PyEntityMailBox();
		/**
	 *	Get a stream for the remote method to add arguments to. 
	 *
	 *	@param methodDesc	The method description.
	 *	@param pHandler		If the method requires a request, this is the
	 *						reply handler to use.
	 */
	virtual BinaryOStream * getStream( const MethodDescription & methodDesc, 
			std::auto_ptr< Mercury::ReplyMessageHandler > pHandler =
				std::auto_ptr< Mercury::ReplyMessageHandler >() ) = 0;
	static PyObject * constructFromRef( const EntityMailBoxRef & ref );
	static bool reduceToRef( PyObject * pObject, EntityMailBoxRef * pRefOutput );

	virtual EntityID id() const = 0;
	virtual void address( const Mercury::Address & addr ) = 0;
	virtual const Mercury::Address address() const = 0;

	virtual void migrate() {}

	typedef PyObject * (*FactoryFn)( const EntityMailBoxRef & ref );
	static void registerMailBoxComponentFactory(
		EntityMailBoxRef::Component c, FactoryFn fn,
		PyTypeObject * pType );

	typedef bool (*CheckFn)( PyObject * pObject );
	typedef EntityMailBoxRef (*ExtractFn)( PyObject * pObject );
	static void registerMailBoxRefEquivalent( CheckFn cf, ExtractFn ef );

	PY_RO_ATTRIBUTE_DECLARE( this->id(), id );
	PyObject * pyGet_address();
	PY_RO_ATTRIBUTE_SET( address );
	
	PY_AUTO_METHOD_DECLARE( RETOWN, callMethod, 
		ARG( ScriptString, ARG( ScriptTuple, END ) ) );
	PyObject * callMethod(
		const ScriptString & methodName, const ScriptTuple & arguments  );

	PyObject * callMethod( 
		const MethodDescription * methodDescription,
		const ScriptTuple & args );
	// 省略很多函数声明
};

这里还提供了一个静态函数来通过EntityMailBoxRef创建具体的PyEntityMailBox子类型对象:

/**
 *	Construct a PyEntityMailBox or equivalent from an EntityMailBoxRef.
 *	Returns Py_None on failure.
 */
PyObject * PyEntityMailBox::constructFromRef(
	const EntityMailBoxRef & ref )
{
	if (ref.id == 0) Py_RETURN_NONE;

	if (s_pRefReg == NULL) Py_RETURN_NONE;

	Fabricators::iterator found = s_pRefReg->fabs_.find( ref.component() );
	if (found == s_pRefReg->fabs_.end()) Py_RETURN_NONE;

	PyObject * pResult = (*found->second)( ref );

	if (pResult)
	{
		return pResult;
	}
	else
	{
		WARNING_MSG( "PyEntityMailBox::constructFromRef: "
				"Could not create mailbox from id %d. addr %s. component %d\n",
				ref.id, ref.addr.c_str(), ref.component() );
		Py_RETURN_NONE;
	}
}

/**
 *	Register a PyEntityMailBox factory
 */
void PyEntityMailBox::registerMailBoxComponentFactory(
	EntityMailBoxRef::Component c, FactoryFn fn, PyTypeObject * pType )
{
	if (s_pRefReg == NULL) s_pRefReg = new MailBoxRefRegistry();
	s_pRefReg->fabs_.insert( std::make_pair( c, fn ) );
	s_pRefReg->mailBoxTypes_.push_back( pType );
}

创建子类型实例的时候使用了一个map来接收所有注册过来的子类型，目前的代码里使用的是手动注册的方式:

/**
 *	This class registers our classes into the PyEntityMailBox system,
 *	and provides some glue/helper functions for it.
 */
static class CellAppPostOfficeAttendant
{
public:
	CellAppPostOfficeAttendant()
	{
		PyEntityMailBox::registerMailBoxComponentFactory(
			EntityMailBoxRef::CELL, newCellMB, &CellEntityMailBox::s_type_ );
		PyEntityMailBox::registerMailBoxComponentFactory(
			EntityMailBoxRef::SERVICE, newBaseMB, &BaseEntityMailBox::s_type_ );
		PyEntityMailBox::registerMailBoxComponentFactory(
			EntityMailBoxRef::BASE, newBaseMB, &BaseEntityMailBox::s_type_ );
		PyEntityMailBox::registerMailBoxComponentFactory(
			EntityMailBoxRef::BASE_VIA_CELL, newBaseViaCellMB, &BaseViaCellMailBox::s_type_ );
		PyEntityMailBox::registerMailBoxComponentFactory(
			EntityMailBoxRef::CELL_VIA_BASE, newCellViaBaseMB, &CellViaBaseMailBox::s_type_ );
		PyEntityMailBox::registerMailBoxComponentFactory(
			EntityMailBoxRef::CLIENT_VIA_CELL, newClientViaCellMB, &ClientViaCellMailBox::s_type_ );
		PyEntityMailBox::registerMailBoxComponentFactory(
			EntityMailBoxRef::CLIENT_VIA_BASE, newClientViaBaseMB, &ClientViaBaseMailBox::s_type_ );
		PyEntityMailBox::registerMailBoxRefEquivalent(
			ServerEntityMailBox::Check, ServerEntityMailBox::static_ref );
		PyEntityMailBox::registerMailBoxRefEquivalent(
			Entity::Check, cellReduce );
	}
	// 省略很多代码
};

可以看出对于之前定义的枚举类EntityMailBoxRef::Component的每个值，都会有一个具体的PyEntityMailBox来承接功能。这里我们简单的来看看使用最多的CellViaBaseMailBox类型:

/**
 *	This class implements a mailbox that can send to a server object. This
 *	object may be on a cell or may be a base.
 *
 *	@see CellEntityMailBox
 *	@see BaseEntityMailBox
 */
class ServerEntityMailBox: public PyEntityMailBox
{
	Py_Header( ServerEntityMailBox, PyEntityMailBox )

public:
	ServerEntityMailBox( EntityTypePtr pBaseType,
			const Mercury::Address & addr, EntityID id,
			PyTypeObject * pType = &s_type_ );
	virtual ~ServerEntityMailBox();

	virtual const Mercury::Address		address() const		{ return addr_; }
	virtual void address( const Mercury::Address & addr )	{ addr_ = addr; }
	virtual void migrate();

	virtual EntityID			id() const			{ return id_; }

	// 省略一些python交互代码
	EntityMailBoxRef ref() const;
	virtual EntityMailBoxRef::Component component() const = 0;
	const char * componentName() const;

	static EntityMailBoxRef static_ref( PyObject * pThis )
		{ return ((const ServerEntityMailBox*)pThis)->ref(); }

	static void migrateMailBoxes();
	static void adjustForDeadBaseApp( const Mercury::Address & deadAddr,
			const BackupHash & backupHash );

protected:

	Mercury::Address			addr_;
	EntityID					id_;

	EntityTypePtr	pLocalType_;
};

/**
 *	This class is common to all mailboxes that send to the base entity or via
 *	the base entity.
 */
class CommonBaseEntityMailBox : public ServerEntityMailBox
{
	Py_Header( CommonBaseEntityMailBox, ServerEntityMailBox )

public:
	CommonBaseEntityMailBox( EntityTypePtr pBaseType,
			const Mercury::Address & addr, EntityID id,
			PyTypeObject * pType = &s_type_ ) :
		ServerEntityMailBox( pBaseType, addr, id, pType )
	{}
	void sendStream();

protected:
	Mercury::Bundle & bundle() const;

private:
	virtual Mercury::UDPChannel & channel() const;
	Mercury::UDPChannel & channel( Entity * pEntity ) const;
};

/**
 *	This class is used to create a mailbox to a cell entity. Traffic for the
 *	entity is sent via the base entity instead of directly to the cell entity.
 *	This means that these mailboxes do not have the restrictions that normal
 *	cell entity mailboxes have.
 */
class CellViaBaseMailBox : public CommonBaseEntityMailBox
{
	Py_Header( CellViaBaseMailBox, CommonBaseEntityMailBox )

	public:
		CellViaBaseMailBox( EntityTypePtr pBaseType,
					const Mercury::Address & addr, EntityID id,
					PyTypeObject * pType = &s_type_ ):
			CommonBaseEntityMailBox( pBaseType, addr, id, pType )
		{}

		~CellViaBaseMailBox() { }

		virtual ScriptObject pyGetAttribute( const ScriptString & attrObj );
		virtual BinaryOStream * getStream( const MethodDescription & methodDesc,
			std::auto_ptr< Mercury::ReplyMessageHandler > pHandler );
		virtual EntityMailBoxRef::Component component() const;
		virtual const MethodDescription * findMethod( const char * attr ) const;
};

没想到这个CellViaBaseMailBox的继承层级还比较深，其继承链为CommonBaseEntityMailBox => ServerEntityMailBox => PyEntityMailBox。在ServerEntityMailBox类型上负责提供Address,EntityId, EntityTypePtr这三个成员变量，基本对应上了EntityMailBoxRef的各种分量，所以这里提供了ref函数来构造EntityMailBoxRef：

/**
 *	Get a packed ref representation of this mailbox
 */
EntityMailBoxRef ServerEntityMailBox::ref() const
{
	EntityMailBoxRef mbr; mbr.init(
		id_, addr_, this->component(), pLocalType_->description().index() );
	return mbr;
}

然后在CommonBaseEntityMailBox这个类型上开始提供bundle和channel这两个接口。这里的channel就是一个信道的概念，负责消息的可靠收发。其实这里的channel并没有绑定在当前的CommonBaseEntityMailBox,而是绑定在RealEntity上，如果当前CellApp里找不到这个RealEntity则使用裸地址来创建channel:

/**
 *  This method returns the most appropriate channel for this mailbox.  The
 *  entity is expected to have already been looked up.  It will use the entity
 *  channel if it can, otherwise it just falls through to the base
 *  implementation.
 */
Mercury::UDPChannel & CommonBaseEntityMailBox::channel( Entity * pEntity ) const
{
	return (pEntity && pEntity->isReal()) ?
		pEntity->pReal()->channel() : CellApp::getChannel( addr_ );
}

然后bundle是单个消息包的概念, 每次调用bundle的时候都会自动的从对应的channel里添加一个新的bundle，然后返回这个bundle的引用来做数据填充:

/**
 *  This method returns a bundle that will be sent to the base entity.  It
 *  overrides the base behaviour of just returning the channel's bundle by
 *  prefixing the bundle with a setClient message if we're not sending on the
 *  entity channel.
 */
Mercury::Bundle & CommonBaseEntityMailBox::bundle() const
{
	Entity * pEntity = CellApp::instance().findEntity( id_ );
	Mercury::Bundle & bundle = this->channel( pEntity ).bundle();

	if (!pEntity || !pEntity->isReal())
	{
		BaseAppIntInterface::setClientArgs::start( bundle ).id = id_;
	}

	return bundle;
}

这里的逻辑是：如果没有找到RealEntity，这会手动的将bundle里填充id字段为要通信的Base对应的EntityID。这就是一个最基本的数据流转换操作。

裸的bundle使用起来过于自由，因为什么都可以往里面填，对于业务层维护消息类型的时候不怎么友好。所以在CellViaBaseMailBox这个类型上提供了getStream接口，这个接口负责初始化一个指定RPC的Bundle:

BinaryOStream * CellViaBaseMailBox::getStream(
		const MethodDescription & methodDesc,
		std::auto_ptr< Mercury::ReplyMessageHandler > pHandler )
{
	Mercury::Bundle & bundle = this->bundle();

	// Not supporting return values
	if (pHandler.get())
	{
		PyErr_Format( PyExc_TypeError,
				"Cannot call two-way method '%s' from CellApp",
				methodDesc.name().c_str() );
		return NULL;
	}

	bundle.startMessage( BaseAppIntInterface::callCellMethod );
	bundle << methodDesc.internalIndex();

	return &bundle;
}

这里负责先使用startMessage来填充当前的消息上层类型为BaseAppIntInterface::callCellMethod，然后再填入methodDesc.internalIndex()代表内层包裹的消息的具体索引。后续业务层再往这个bundle填充消息的时候，完全不知道这个RPC消息其实已经被添加了一个BaseAppIntInterface::callCellMethod的头部，神不知鬼不觉的做了一个消息转换。

数据流与封包

在Bigworld里，往bundle里填充数据使用的是类似于std::iostream的流式处理，利用operator<<来填入数据，然后用operator>>来解析数据。在bundle的头文件声明里，已经提供好了绝大部分的基础类型的operator<<的支持，例如bool,int,String,Vector等:

inline BinaryOStream& operator<<( BinaryOStream &out, int64 x )
{
	bw_netlonglong n;
	n.i64 = x;
	return out << n;
}

inline BinaryOStream& operator<<( BinaryOStream &out, char x )
{
	bw_netbyte n;
	n.c = x;
	return out << n;
}


/**
 *	This method provides output streaming for a string.
 *
 *	@param b	The binary stream.
 *	@param str	The string to be streamed.
 *
 *	@return A reference to the stream that was passed in.
 */
inline BinaryOStream & operator<<( BinaryOStream & b, const BW::string & str )
{
	b.appendString( str.data(), int(str.length()) );

	return b;
}

/**
 * 	This method provides output streaming for a vector.
 *
 *	@param b	The binary stream.
 *	@param data	The vector to be streamed.
 *
 *	@return A reference to the stream that was passed in.
 */
template <class T, class A>
inline BinaryOStream & operator<<( BinaryOStream & b,
		const BW::vector<T,A> & data)
{
	uint32 max = (uint32)data.size();
	b << max;
	for (uint32 i=0; i < max; i++)
		b << data[i];
	return b;
}
class Bundle : public BinaryOStream
{
	// 省略类型的具体实现
}；

不过这里没有提供对Map等复杂容器类型的支持，所以上面的CellAppInterface::addCell样例代码在序列化一个Map的时候，需要手动的序列化。下面就是一个手动序列化容器的实例，先序列化当前Map的大小，然后对内部的每个Pair执行逐个基础元素的序列化:

struct DataEntry
{
	uint16 key;
	BW::string data;
};
typedef BW::map< SpaceEntryID, DataEntry > DataEntries;
DataEntries dataEntries_;

{
	// bundle.startMessage( CellAppInterface::allSpaceData );
	// bundle << id_;
	bundle << (uint32)dataEntries_.size();

	DataEntries::const_iterator iter = dataEntries_.begin();

	while (iter != dataEntries_.end())
	{
		bundle << iter->first <<
			iter->second.key << iter->second.data;
		++iter;
	}
}

当逻辑层接收到这个RPC的时候，需要按照之前执行operator<<的顺序来执行operator>>，从而达到正确的参数解析:

/**
 *	This method reads space data from the input stream.
 */
void Space::readDataFromStream( BinaryIStream & stream )
{
	int size;
	stream >> size;

	for (int i = 0; i < size; i++)
	{
		SpaceEntryID entryID;
		uint16 key;
		BW::string value;
		stream >> entryID >> key >> value;

		this->spaceDataEntry( entryID, key, value, DONT_UPDATE_CELL_APP_MGR );
	}
}

解释了数据流是如何序列化与反序列化的之后，我们再来看一下Bundle里是如何执行封包逻辑的，这里只讲解UDPBundle，因为这个子类用的最多。封包逻辑入口是startMessage，参数为一个RPC的描述元数据InterfaceElement,以及是否是可靠消息的标记位，我们目前先默认全都是可靠的：

/**
 * 	This method starts a new message on the bundle.
 *
 * 	@param ie			The type of message to start.
 * 	@param reliable		True if the message should be reliable.
 */
void UDPBundle::startMessage( const InterfaceElement & ie, 
		ReliableType reliable )
{
	// Piggybacks should only be added immediately before sending.
	MF_ASSERT( !pCurrentPacket_->hasFlags( Packet::FLAG_HAS_PIGGYBACKS ) );
	MF_ASSERT( ie.name() );

	this->endMessage();
	curIE_ = ie;
	msgIsReliable_ = reliable.isReliable();
	msgIsRequest_ = false;
	isCritical_ = (reliable == RELIABLE_CRITICAL);
	this->newMessage();

	reliableDriver_ |= reliable.isDriver();
}

这里会先使用endMessage来完成之前的逻辑包的封装工作，然后使用newMessage来开启一个新逻辑包。endMessage的具体内容我们先不看，先看一下newMessage在干什么。函数开头先对包的一些统计信息进行修改，然后使用qreserve来预留相应大小的buffer来供后续的参数来填充:

/**
 * 	This message begins a new message, with the given number of extra bytes in
 * 	the header. These extra bytes are normally used for request information.
 *
 * 	@param extra	Number of extra bytes to reserve.
 * 	@return	Pointer to the body of the message.
 */
char * UDPBundle::newMessage( int extra )
{
	// figure the length of the header
	int headerLen = curIE_.headerSize();
	if (headerLen == -1)
	{
		CRITICAL_MSG( "Mercury::UDPBundle::newMessage: "
			"tried to add a message with an unknown length format %d\n",
			(int)curIE_.lengthStyle() );
	}

	++numMessages_;

	if (msgIsReliable_)
	{
		++numReliableMessages_;
	}

	// make space for the header
	MessageID * pHeader = (MessageID *)this->qreserve( headerLen + extra );

	// set the start of this msg
	msgBeg_ = (uint8*)pHeader;
	msgChunkOffset_ = Packet::Offset( pCurrentPacket_->msgEndOffset() );

	// write in the identifier
	*(MessageID*)pHeader = curIE_.id();

	// set the length to zero
	msgLen_ = 0;
	msgExtra_ = extra;

	// and return a pointer to the extra data
	return (char *)(pHeader + headerLen);
}

这里的pHeader就是当前消息的buffer开始地址，这里先往buffer的第一个字节填充进入当前消息的类型id，然后返回Header结束后的地址，这里预留buffer的接口是qreserve。之前在基类BinaryOStream上的所有operator<<最终都会通过reserve这个虚接口来将数据写入到buffer中， 在UDPBundle里这个虚接口的实现就是qreserve::

inline BinaryOStream& operator<<( BinaryOStream &out, bw_netlong x )
{
	BW_STATIC_ASSERT( sizeof( bw_netlong ) == 4, bw_netlong_bad_size );
	*(uint32*)out.reserve( sizeof( x ) ) = BW_HTONL( x.u32 );
	return out;
}

inline BinaryOStream& operator<<( BinaryOStream &out, bw_netlonglong x )
{
	BW_STATIC_ASSERT( sizeof( bw_netlonglong ) == 8, bw_netlonglong_bad_size );
	*(uint64*)out.reserve( sizeof( x ) ) = BW_HTONLL( x.u64 );
	return out;
}
/**
 * 	This method reserves the given number of bytes in this bundle.
 */
INLINE void * UDPBundle::reserve( int nBytes )
{
	return qreserve( nBytes );
}

所以这个qreserve就是序列化中最重要的函数，负责提供足够大的buffer来填入后续数据，我们来看看这个qreserve是如何执行动态内存分配的:

/**
 * 	This method gets a pointer to this many bytes quickly
 * 	(non-virtual function)
 */
INLINE void * UDPBundle::qreserve( int nBytes )
{
	if (nBytes <= pCurrentPacket_->freeSpace())
	{
		void * writePosition = pCurrentPacket_->back();
		pCurrentPacket_->grow( nBytes );
		return writePosition;
	}
	else
	{
		return this->sreserve( nBytes );
	}
}

从这里可以看出，Bundle的更底层组成单位为Packet，是作为一段连续内存buffer而存在的，pCurrentPacket_就是当前正在被使用的buffer。如果当前pCurrentPacket_的剩余空间不满足申请的大小，则使用sreserve来结束当前Packet,并新建一个Packet来作为新的buffer:

/**
 *  This function returns a pointer to nBytes on a bundle.
 *  It assumes that the data will not fit in the current packet,
 *  so it adds a new one. This is a private function.
 *
 *  @param nBytes	Number of bytes to reserve.
 *
 *  @return	Pointer to the reserved data.
 */
void * UDPBundle::sreserve( int nBytes )
{
	this->endPacket( /* isExtending */ true );
	this->startPacket( new Packet() );

	void * writePosition = pCurrentPacket_->back();
	pCurrentPacket_->grow( nBytes );

	MF_ASSERT( pCurrentPacket_->freeSpace() >= 0 );
	return writePosition;
}

Packet内部使用一个固定大小PACKET_MAX_SIZE的数组作为底层buffer，这里的Packet::back返回的是剩下还没有使用的内存开始地址，grow的作用就是将back后面的nBytes标记为已经使用，同时将back后移:

#define PACKET_MAX_SIZE 1472
/**
 *	All packets look like this. Only the data is actually sent;
 *	the rest is just housekeeping.
 *
 *	@ingroup mercury
 */
class Packet : public ReferenceCount
{
	public:
	Packet();
	~Packet();

	Packet * next()				{ return next_.get(); }
	const Packet * next() const	{ return next_.get(); }

	void chain( Packet * pPacket ) { next_ = pPacket; }
public:	
	char * data() { return data_; }
	const char * data() const { return data_; }

	/// Returns a pointer to the start of the message data.
	const char * body() const { return data_ + HEADER_SIZE; }

	/// Returns a pointer to the end of the message data.
	char * back() { return data_ + msgEndOffset_; }

	int msgEndOffset() const	{ return msgEndOffset_; }
	int bodySize() const		{ return msgEndOffset_ - HEADER_SIZE; }
	int footerSize() const		{ return footerSize_; }
	int totalSize() const		{ return msgEndOffset_ + footerSize_; }

	void msgEndOffset( int offset )		{ msgEndOffset_ = offset; }
	void grow( int nBytes )				{ msgEndOffset_ += nBytes; }
	void shrink( int nBytes )			{ msgEndOffset_ -= nBytes; }

	int freeSpace() const
	{
		return MAX_SIZE -
			RESERVED_FOOTER_SIZE -
			msgEndOffset_ -
			footerSize_ -
			extraFilterSize_;
	}
private:
	/// Packets are linked together in a simple linked list fashion.
	PacketPtr	next_;

	/// This the offset of the end of the headers and message data. It is
	/// temporarily incorrect in two situations: when sending, it is incorrect
	/// in NetworkInterface::send() whilst footers are being written, and when
	/// receiving, it  is incorrect until processOrderedPacket() strips the
	/// fragment footers.
	int			msgEndOffset_;
	/// The variable-length data follows the packet header in memory.
	char			data_[PACKET_MAX_SIZE];
};

然后从这个类型声明可以看出，连续的Packet是使用next_指针相连，组成一个单链表。因此在startPacket的时候，负责使用Packet::chain来将新的Packet拼接到当前pCurrentPacket_的后面，然后将pCurrentPacket_更新为当前新分配的packet:

/**
 *  This method starts a new packet in this bundle.
 */
void UDPBundle::startPacket( Packet * p )
{
	Packet * prevPacket = pCurrentPacket_;

	// Link the new packet into the chain if necessary.
	if (prevPacket)
	{
		prevPacket->chain( p );
	}

	pCurrentPacket_ = p;
	pCurrentPacket_->reserveFilterSpace( extraSize_ );

	pCurrentPacket_->setFlags( 0 );

	pCurrentPacket_->msgEndOffset( Packet::HEADER_SIZE );

	// if we're in the middle of a message start the next chunk here
	msgChunkOffset_ = pCurrentPacket_->msgEndOffset();
}

虽然我们在Packet::data里预留了PACKET_MAX_SIZE个字节，但是其实里面真正可用的比这个小，因为要扣除Packet::HEADER_SIZ, Packet::RESERVED_FOOTER_SIZE和FilterSpace等数据。总的来说这里会控制单个Packet的实际数据大小要小于1400，刚好是以太网的常见MTU大小。

如果当前要填充的数据很大，导致单一Packet放不下会怎么办？解决方式是使用多个Packet串联起来，例如这里的添加大量二进制数据的接口addBlob会使用循环来进行可能的多次Packet分配:

/**
 *	This convenience method is used to add a block of memory to this stream.
 */
INLINE
void UDPBundle::addBlob( const void * pBlob, int size )
{
	const char * pCurr = (const char *)pBlob;

	while (size > 0)
	{
		// If there isn't any more space on this packet, force a new one to be
		// allocated to this bundle.
		if (pCurrentPacket_->freeSpace() == 0)
		{
			this->sreserve( 0 );
		}

		int currSize = std::min( size, int( pCurrentPacket_->freeSpace() ) );
		MF_ASSERT( currSize > 0 );

		memcpy( this->qreserve( currSize ), pCurr, currSize );
		size -= currSize;
		pCurr += currSize;
	}
}

了解了Packet是如何填充的之后，我们再来回顾一下每次StartPacket之前都需要先执行的endPacket，其实就是记录一些统计信息：

/**
 *	This method end processing of the current packet, i.e. calculate its
 *	flags, and the correct size including footers.
 *
 *	@param isExtending	True if we are extending the bundle size, false
 *						otherwise (when we are finalising for send).
 */
void UDPBundle::endPacket( bool isExtending )
{
	// If this won't be the last packet, add a reliable order marker
	if (isExtending)
	{
		if (this->isOnExternalChannel())
		{
			// add a partial reliable order if in the middle of a message
			if (msgBeg_ != NULL && msgIsReliable_)
			{
				this->addReliableOrder();
			}

			// add a gap reliable order to mark the end of the packet
			ReliableOrder rgap = { NULL, 0, 0 };
			reliableOrders_.push_back( rgap );
		}
	}

	// if we're in the middle of a message add this chunk
	msgLen_ += pCurrentPacket_->msgEndOffset() - msgChunkOffset_;
	msgChunkOffset_ = uint16( pCurrentPacket_->msgEndOffset() );
}

当一个逻辑消息包Message彻底填充结束的时候，会调用endMessage这个接口。在这个函数里会使用compressLength将当前Message的总长度msgLen_写入到当前消息字节流的开头，这部分空间在StartMessage的时候已经预留：

/**
 * 	This method finalises a message. It is called from a number of places
 *	within Bundle when necessary.
 */
void UDPBundle::endMessage( bool isEarlyCall /* = false */ )
{
	// nothing to do if no message yet
	if (msgBeg_ == NULL)
	{
		MF_ASSERT( pCurrentPacket_->msgEndOffset() == Packet::HEADER_SIZE || 
			hasEndedMsgEarly_ );
		return;
	}

	// add the amt used in this packet to the length
	msgLen_ += pCurrentPacket_->msgEndOffset() - msgChunkOffset_;

	// fill in headers for this msg
	curIE_.compressLength( msgBeg_, msgLen_, this, msgIsRequest_ );

	// record its details if it was reliable
	if (msgIsReliable_)
	{
		if (this->isOnExternalChannel())
		{
			this->addReliableOrder();
		}

		msgIsReliable_ = false;	// for sanity
	}

	msgChunkOffset_ = Packet::Offset( pCurrentPacket_->msgEndOffset() );

	msgBeg_ = NULL;
	msgIsRequest_ = false;

	hasEndedMsgEarly_ = isEarlyCall;
}

这里的compressLength逻辑并不是使用一个固定的uint32_t的空间来填入当前消息包的总长度，而是使用了变长编码。他根据当前RPC接口所携带的长度信息来做这样的处理：

1. 如果当前RPC是参数大小固定的RPC，则不需要在包开头预留长度字段，无需填充
2. 如果当前RPC是参数大小可变的RPC，则根据预设的参数最大长度的字节大小lengthParam_来在包开头预留对应的空间，等一个包所有参数彻底填入之后，再在开头预留的空间里将这个长度字段填进入
3. 如果当前RPC是参数大小可变的RPC，但是当前参数的大小超过了之前预设的lengthParam_所能表示的整数范围，则将预先保留的大小区域全都填充为0xff，然后在bundle的末尾再加入一个int32，写入当前包大小之后，再与第一个packet里的头四个字节做调换，这样长度字段依然在message的开头部分，这部分对应的代码见下：

// If the message length could not fit into a standard length field, we
// need to handle this as a special case.
if (oversize)
{
	// Fill the original length field with ones to indicate the special
	// situation.
	static const int IDENTIFIER_SIZE = sizeof(uint8);
	for (int i = IDENTIFIER_SIZE; i <= lengthParam_; ++i)
	{
		((uint8*)data)[i] = 0xff;
	}

	if (pBundle)
	{
		void * tail = pBundle->reserve( sizeof( int32 ) );
		void * ret = this->specialCompressLength( data, length,
			pBundle->pFirstPacket(), isRequest );
		MF_ASSERT( !ret || tail == ret );
		return ret ? 0 : -1;
	}
	else
	{
		return -1;
	}
}

/**
 *	This method is called by InterfaceElement::compressLength when the amount
 *	of data added to the stream for the message is more than the message's size
 *	field can handle. For example, if lengthParam is 1 and there is at least
 *	255 bytes worth of data added for the message (or 65535 for 2 bytes etc).
 *
 *	To handle this, a 4-byte size is placed at the start of the message
 *	displacing the first four bytes of the message. These are appended to the
 *	end of the message. The original length field is filled with 0xff to
 *	indicate this special situation.
 */
void * InterfaceElement::specialCompressLength( void * data, int length,
		Packet * pPacket, bool isRequest ) const;

对应这样的长度填充函数compressLength，InterfaceElement也提供了对应的长度解析函数expandLength，就是当前填充过程的逆过程:

/**
 * 	This method expands a length from the given header.
 *
 * 	@param data	This is a pointer to a message header.
 *	@param pPacket
 *	@param isRequest
 *
 * 	@return Expanded length.
 */
int InterfaceElement::expandLength( void * data, Packet * pPacket, 
		bool isRequest ) const
{
	switch (lengthStyle_)
	{
	case FIXED_LENGTH_MESSAGE:
		return lengthParam_;
		break;
	case VARIABLE_LENGTH_MESSAGE:
	{
		uint8 *pLen = ((uint8*)data) + sizeof( MessageID );
		uint32 len = 0;

		switch (lengthParam_)
		{
			case 0: len = 0; break;
			case 1: len = *(uint8*)pLen; break;
			case 2:
			{
#if defined( BW_ENFORCE_ALIGNED_ACCESS )
				uint16 len16 = 0;
				memcpy( &len16, pLen, sizeof(uint16) );
				len = BW_NTOHS( len16 );
#else // !defined( BW_ENFORCE_ALIGNED_ACCESS )
				len = BW_NTOHS( *(uint16 *)pLen );
#endif // defined( BW_ENFORCE_ALIGNED_ACCESS )
				break;
			}
			case 3: len = BW_UNPACK3( (const char*)pLen ); break;
			case 4:
			{
#if defined( BW_ENFORCE_ALIGNED_ACCESS )
				uint32 len32;
				memcpy( &len32, pLen, sizeof(uint32) );
				len = BW_NTOHL( len32 );
#else // !defined( BW_ENFORCE_ALIGNED_ACCESS )
				len = BW_NTOHL( *(uint32*)pLen );
#endif // defined( BW_ENFORCE_ALIGNED_ACCESS )
				break;
			}
			default:
				CRITICAL_MSG( "InterfaceElement::expandLength( %s ): "
					"Unhandled variable message length: %d\n",
					this->c_str(), lengthParam_ );
		}

		// If lengthParam_ is 4, a length > 0x80000000 will cause an overflow
		// and a negative value will be returned from this method.
		if ((int)len < 0)
		{
			ERROR_MSG( "Mercury::InterfaceElement::expandLength( %s ): "
				"Overflow in calculating length of variable message!\n",
				this->c_str() );

			return -1;
		}

		// The special case is indicated with the length field set to maximum.
		// i.e. All bits set to 1.
		if (!this->canHandleLength( len ))
		{
			if (!pPacket)
			{
				return -1;
			}
			return this->specialExpandLength( data, pPacket, isRequest );
		}

		return len;
		break;
	}
	default:
		ERROR_MSG( "Mercury::InterfaceElement::expandLength( %s ): "
			"unrecognised length format %d\n",
			this->c_str(), (int)lengthStyle_ );

		break;
	}
	return -1;
}

综上, bundle里的最小组成单元其实是packet，packet之间使用单链表来串联。每个packet的预留大小都是一样的，基本等于ip的最常见MTU 1400。然后每个Message的所有数据都放在packet里，每个Message的开头都会写入一个单字节整数代表当前Message的唯一标识符。在标识符后面就是当前Message的参数总长度信息，这是一个可变字节大小的整数，字节大小由这个Message的类型确定。如果消息太长，则会走上面的oversize逻辑来填入长度信息。

Bundle的发送与接收

Message写入之后并不会立即触发底层的网络发送，网络发送的逻辑由Channel::send托管,一般是每帧末尾被自动调用，也可以手动调用:

/**
 *	This method sends the given bundle on this channel. If no bundle is
 *	supplied, the channel's own bundle will be sent.
 *
 *	@param pBundle 	The bundle to send, or NULL to send the channel's own
 *					bundle.
 */
void Channel::send( Bundle * pBundle /* = NULL */ )
{
	ChannelPtr pChannel( this );

	if (!this->isConnected())
	{
		ERROR_MSG( "Channel::send( %s ): Channel is not connected\n",
			this->c_str() );
		return;
	}

	if (pBundle == NULL)
	{
		pBundle = pBundle_;
	}

	this->doPreFinaliseBundle( *pBundle );

	pBundle->finalise();

	this->networkInterface().addReplyOrdersTo( *pBundle, this );

	this->doSend( *pBundle );

	// Clear the bundle
	if (pBundle == pBundle_)
	{
		this->clearBundle();
	}
	else
	{
		pBundle->clear();
	}

	if (pListener_)
	{
		pListener_->onChannelSend( *this );
	}
}

这里的核心就是doSend函数，内部会做一堆错误检查，最后没有问题的情况下才会调用到networkInterface::send:

/*
 *	Override from Channel.
 */
void UDPChannel::doSend( Bundle & bundleUncast )
{
	MF_ASSERT_DEBUG( dynamic_cast< UDPBundle * >( &bundleUncast ) != NULL );
	UDPBundle * pBundle = static_cast< UDPBundle * >( &bundleUncast );

	// 忽略很多错误处理函数

	// Send the bundle through the network interface as UDP packets
	pNetworkInterface_->send( addr_, *pBundle, this );

	// Update our stats
	++numDataUnitsSent_;
	numBytesSent_ += pBundle->size();

	if (pBundle->isReliable())
	{
		++numReliablePacketsSent_;
	}

	// Channels that do not send regularly are added to a collection to do
	// their resend checking periodically.
	pNetworkInterface_->irregularChannels().addIfNecessary( *this );

	// If the bundle that was just sent was critical, the sequence number of
	// its last packet is the new unackedCriticalSeq_.
	if (pBundle->isCritical())
	{
		unackedCriticalSeq_ =
			pBundle->pFirstPacket()->seq() + pBundle->numDataUnits() - 1;
	}
}

/**
 * 	This method sends a bundle to the given address.
 *
 * 	Note: any pointers you have into the packet may become invalid after this
 * 	call (and whenever a channel calls this too).
 *
 * 	@param address	The address to send to.
 * 	@param bundle	The bundle to send
 *	@param pChannel	The Channel that is sending the bundle.
 *				(even if the bundle is not sent reliably, it is still passed
 *				through the filter associated with the channel).
 */
void NetworkInterface::send( const Address & address,
								UDPBundle & bundle, UDPChannel * pChannel )
{
	pPacketSender_->send( address, bundle, pChannel );
}

这里有一个专门的PacketSender来负责组包并投递到底层的socket里:

/**
 * 	This method sends a bundle to the given address.
 *
 * 	Note: any pointers you have into the packet may become invalid after this
 * 	call (and whenever a channel calls this too).
 *
 * 	@param address	The address to send to.
 * 	@param bundle	The bundle to send
 *	@param pChannel	The Channel that is sending the bundle.
 *				(even if the bundle is not sent reliably, it is still passed
 *				through the filter associated with the channel).
 */
void PacketSender::send( const Address & address,
		UDPBundle & bundle, UDPChannel * pChannel )
{
	MF_ASSERT( address != Address::NONE );
	MF_ASSERT( !pChannel || pChannel->addr() == address );

	MF_ASSERT( !bundle.pChannel() || (bundle.pChannel() == pChannel) );

#if ENABLE_WATCHERS
	sendingStats_.mercuryTimer().start();
#endif // ENABLE_WATCHERS

	if (!bundle.isFinalised())
	{
		// Handle bundles sent off-channel that won't have been finalised by
		// their channels yet.
		bundle.finalise();
		bundle.addReplyOrdersTo( &requestManager_, pChannel );
	}

	// fill in all the footers that are left to us
	Packet * pFirstOverflowPacket = bundle.preparePackets( pChannel,
				seqNumAllocator_, sendingStats_, shouldUseChecksums_ );

	// Finally actually send the darned thing. Do not send overflow packets.
	for (Packet * pPacket = bundle.pFirstPacket();
			pPacket != pFirstOverflowPacket;
			pPacket = pPacket->next())
	{
		this->sendPacket( address, pPacket, pChannel, false );
	}

#if ENABLE_WATCHERS
	sendingStats_.mercuryTimer().stop( 1 );
#endif // ENABLE_WATCHERS

	sendingStats_.numBundlesSent_++;
	sendingStats_.numMessagesSent_ += bundle.numMessages();

	sendingStats_.numReliableMessagesSent_ += bundle.numReliableMessages();
}

在这里我们看到有一个for循环，在循环里会把当前单链表里的Packet依次的调用sendPacket来投入到socket里。但是这里有一个限制，不能单词send调用发送太多数据，因此会使用bundle.preparePackets来计算出最多能发送到的Packet记录为pFirstOverflowPacket，循环遍历的时候遇到这个Packet就结束。同时preparePackets还会将当前Channel的唯一标识符(其实就是EntityID)也加入到Packet里:

/**
 *  This method will write the flags on a packet fitting for one that will ride
 *  on this channel. It will also reserve enough space for the footer.
 */
void UDPChannel::writeFlags( Packet * p )
{
	p->enableFlags( Packet::FLAG_ON_CHANNEL );

	if (this->isIndexed())
	{
		p->enableFlags( Packet::FLAG_INDEXED_CHANNEL );
		p->channelID() = id_;
		p->channelVersion() = version_;
		p->reserveFooter( sizeof( ChannelID ) + sizeof( ChannelVersion ) );
	}
	// 省略很多代码
}

这里的sendPacket还是过于上层，会有很多的统计和中转过滤操作，并最终调用到basicSendWithRetries:

/**
 *	This method sends a packet. No result is returned as it cannot be trusted.
 *	The packet may never get to the other end.
 *
 *	@param address 	The destination address.
 *	@param pPacket	The packet to send.
 *	@param pChannel The channel to be sent on, or NULL if off-channel.
 *	@param isResend If true, this is a resend, otherwise, it is the initial
 *					send.
 */
void PacketSender::sendPacket( const Address & address,
						Packet * pPacket,
						UDPChannel * pChannel, bool isResend )
{
	// 忽略一些无关代码

	// Check if we want artificial loss or latency
	if (!this->rescheduleSend( address, pPacket, pChannel ))
	{
		this->sendRescheduledPacket( address, pPacket, pChannel );
	}
}

/**
 *	This method sends the packet after rescheduling has occurred.
 *
 *	@param address 	The destination address.
 *	@param pPacket 	The packet.
 *	@param pChannel The channel, or NULL if off-channel send.
 */
void PacketSender::sendRescheduledPacket( const Address & address,
						Packet * pPacket,
						UDPChannel * pChannel )
{
	PacketFilterPtr pFilter = pChannel ? pChannel->pFilter() : NULL;

	if (pPacketMonitor_)
	{
		pPacketMonitor_->packetOut( address, *pPacket );
	}

	// Otherwise send as appropriate
	if (pFilter)
	{
		pFilter->send( *this, address, pPacket );
	}
	else
	{
		this->basicSendWithRetries( address, pPacket );
	}
}

这里的basicSendWithRetries负责封装底层socket的发送，真正的往指定地址尝试投递一个包，如果投递失败还负责短暂的重试:

/**
 *	Basic packet sending functionality that retries a few times
 *	if there are transitory errors.
 *
 *	@param addr 	The destination address.
 *	@param pPacket 	The packet to send.
 *
 *	@return 		REASON_SUCCESS on success, otherwise an appropriate
 *					Mercury::Reason.
 */
Reason PacketSender::basicSendWithRetries( const Address & addr,
		Packet * pPacket )
{
	// try sending a few times
	int retries = 0;
	Reason reason;

	while (retries <= 3)
	{
		++retries;
#if ENABLE_WATCHERS
		sendingStats_.systemTimer().start();
#endif // ENABLE_WATCHERS

		reason = this->basicSendSingleTry( addr, pPacket );

#if ENABLE_WATCHERS
		sendingStats_.systemTimer().stop( 1 );
#endif // ENABLE_WATCHERS

		if (reason == REASON_SUCCESS)
			return reason;

		// If we've got an error in the queue simply send it again;
		// we'll pick up the error later.
		if (reason == REASON_NO_SUCH_PORT)
		{
			continue;
		}

		// If the transmit queue is full wait 10ms for it to empty.
		if ((reason == REASON_RESOURCE_UNAVAILABLE) ||
				(reason == REASON_TRANSMIT_QUEUE_FULL))
		{
			// 一些容错代码
			continue;
		}

		// some other error, so don't bother retrying
		break;
	}

	return reason;
}

最后的basicSendSingleTry才有机会接触到裸的socket，将当前packet里的数据发送到指定socket里，至此发送流程终于走完了：

/**
 *	Basic packet sending function that just tries to send once.
 *
 *	@param addr 	The destination address.
 *	@param pPacket 	The packet to send.
 *
 *	@return 		REASON_SUCCESS on success otherwise an appropriate
 *					Mercury::Reason.
 */
Reason PacketSender::basicSendSingleTry( const Address & addr, 
		Packet * pPacket )
{
	int len = socket_.sendto( pPacket->data(), pPacket->totalSize(), 
		addr.port, addr.ip );

	if (len == pPacket->totalSize())
	{
		sendingStats_.numBytesSent_ += len + UDP_OVERHEAD;
		sendingStats_.numPacketsSent_++;

		return REASON_SUCCESS;
	}
	// 省略错误处理代码
}

Packet的发送由PacketSender托管， 类似的Packet的接收也有一个专门的类型PacketReceiver托管。不过收消息逻辑上层还有很多封装，下面就是一个典型的收消息调用栈，:

WorldOfWarplanes.exe!Mercury::PacketReceiver::processFilteredPacket(const Mercury::Address & addr={...}, Mercury::Packet * p=0x243399c0, Mercury::ProcessSocketStatsHelper * pStatsHelper=0x0018f2a4)  Line 347 + 0x11 bytes C++
WorldOfWarplanes.exe!Mercury::PacketFilter::recv(Mercury::PacketReceiver & receiver={...}, const Mercury::Address & addr={...}, Mercury::Packet * pPacket=0x243399c0, Mercury::ProcessSocketStatsHelper * pStatsHelper=0x0018f2a4)  Line 38 C++
WorldOfWarplanes.exe!Mercury::EncryptionFilter::recv(Mercury::PacketReceiver & receiver={...}, const Mercury::Address & addr={...}, Mercury::Packet * pPacket=0x243399c0, Mercury::ProcessSocketStatsHelper * pStatsHelper=0x0018f2a4)  Line 233 + 0x1c bytes C++
WorldOfWarplanes.exe!Mercury::PacketReceiver::processPacket(const Mercury::Address & addr={...}, Mercury::Packet * p=0x0493cc40, Mercury::ProcessSocketStatsHelper * pStatsHelper=0x0018f2a4)  Line 242 + 0x25 bytes C++
WorldOfWarplanes.exe!Mercury::PacketReceiver::processSocket(bool expectingPacket=true)  Line 92 C++
WorldOfWarplanes.exe!Mercury::PacketReceiver::handleInputNotification(int fd=684)  Line 51 + 0x9 bytes C++
WorldOfWarplanes.exe!Mercury::SelectPoller::handleInputNotifications(int & countReady=, fd_set & readFDs={...}, fd_set & writeFDs={...})  Line 305 + 0x29 bytes C++
WorldOfWarplanes.exe!Mercury::SelectPoller::processPendingEvents(double maxWait=0.00000000000000000)  Line 398 + 0x19 bytes C++
WorldOfWarplanes.exe!Mercury::DispatcherCoupling::doTask()  Line 34 + 0x38 bytes C++
WorldOfWarplanes.exe!Mercury::FrequentTasks::process()  Line 112 C++
WorldOfWarplanes.exe!Mercury::EventDispatcher::processOnce(bool shouldIdle=false)  Line 381 C++
WorldOfWarplanes.exe!ServerConnection::processInput()  Line 922 C++

在PacketReceiver::handleInputNotification(int fd=684)上层的逻辑都是一些epoll的封装，这些封装代码我们这里就跳过。当单个socket可以读取数据的时候才会调用到PacketReceiver::handleInputNotification，这里会不断的调用processSocket来接收一个Packet，直到socket里的剩余可读数据无法组成一个Packet:

/*
 *	This method is called when there is data on the socket.
 */
int PacketReceiver::handleInputNotification( int fd )
{
	uint64 processingStartStamps = BW_NAMESPACE timestamp();

	int numPacketsProcessed = 0;

	bool expectingPacket = true; // only true for first call to processSocket()
	bool shouldProcess = true;

	while (shouldProcess)
	{
		Address sourceAddress;
		shouldProcess = this->processSocket( sourceAddress, expectingPacket );
		expectingPacket = false;

		uint64 processingElapsedStamps = 
			BW_NAMESPACE timestamp() - processingStartStamps;

		++numPacketsProcessed;

		// 省略一些容错代码

	}

	return 0;
}

这个processSocket会通过Packet::recvFromEndpoint来尝试从socket里读取数据来填充当前Packet：

/**
 *	This method will read and process any pending data on this object's socket.
 *
 *	@param srcAddr 			This will be filled with the source address of any
 *							packets received.
 *	@param expectingPacket 	If true, a packet was expected to be read, 
 *							otherwise false.
 *
 *	@return 				True if a packet was read, otherwise false.
 */
bool PacketReceiver::processSocket( Address & srcAddr, 
		bool expectingPacket )
{
	stats_.updateSocketStats( socket_ );

	// Used to collect stats
	ProcessSocketStatsHelper statsHelper( stats_ );

	// try a recvfrom
	int len = pNextPacket_->recvFromEndpoint( socket_, srcAddr );

	statsHelper.socketReadFinished( len );

	if (len <= 0)
	{
		this->checkSocketErrors( len, expectingPacket );
		return false;
	}

	// process it if it succeeded
	PacketPtr curPacket = pNextPacket_;
	pNextPacket_ = new Packet();

	Reason ret = this->processPacket( srcAddr, curPacket.get(),
			&statsHelper );

	if ((ret != REASON_SUCCESS) &&
			networkInterface_.isVerbose())
	{
		this->dispatcher().errorReporter().reportException( ret, srcAddr );
	}

	return true;
}

Packet::recvFromEndpoint每次尝试读取最多PACKET_MAX_SIZE=1472个字节的数据到内部的读取缓冲区内，这个PACKET_MAX_SIZE也就是当前单一Packet的数据大小上限：

// The default max size for a packet is the MTU of an ethernet frame, minus the
// overhead of IP and UDP headers.  If you have special requirements for packet
// sizes (e.g. your client/server connection is running over VPN) you can edit
// this to whatever you need.
const int Packet::MAX_SIZE = PACKET_MAX_SIZE;

/**
 *  This method does a recv on the endpoint into this packet's data array,
 *  setting the length correctly on a successful receive.  The return value is
 *  the return value from the low-level recv() call.
 */
int Packet::recvFromEndpoint( Endpoint & ep, Address & addr )
{
	int len = ep.recvfrom( data_, MAX_SIZE,
		(u_int16_t*)&addr.port, (u_int32_t*)&addr.ip );

	if (len >= 0)
	{
		this->msgEndOffset( len );
	}

	return len;
}

这里需要重新学习一下UDP的数据接收，他不像TCP那样的字节流接收，而是以包为单位来接收，当一个UDP socket变的可读的时候说明其内部已经接收了一个完整的包。业务层recvfrom接收udp包的时候需要传入buf和bufsize，就是接收空间和接收空间大小。如果这个bufsize小于udp包的大小，那么只能接收到这个udp包的前bufsize个字节，剩下的部分会被直接被丢弃，再次执行recvfrom的时候处理的已经是第二个包了。所以bufsize要适配组包时的单Packet大小上限，也就是PACKET_MAX_SIZE。

所以当recvFromEndpoint返回的len大于0的时候，就代表接收到了一个Packet。由于我们业务层使用的是Message，所以接下来我们需要尝试从Packet里恢复出Message：

/**
 *	This is the entrypoint for new packets, which just gives it to the filter.
 */
Reason PacketReceiver::processPacket( const Address & addr, Packet * p,
	   ProcessSocketStatsHelper * pStatsHelper )
{
	// 跳过一些分支处理

	//parse raw.
	return this->processFilteredPacket( addr, p, pStatsHelper );
}

这个processFilteredPacket的内部逻辑太长了，我们这里就跳过，只需要知道的是这里会根据传入的地址addr与当前PacketReceiver绑定的RPC Interface以及Packet负载的ChannelID来查找对应的Channel pChannel，找到之后执行addToReceiveWindow函数:

// 省略很多代码
// should we be looking in a channel
if (pChannel)
{
	UDPChannel::AddToReceiveWindowResult result =
		pChannel->addToReceiveWindow( p, addr, stats_ );
}
// 省略很多代码

由于udp的Packet可能会被乱序接收，所以这里的addToReceiveWindow的作用就是将接受到的Packet按照其编号进行排序，如果当前接收到的Packet的序号与所期待的Packet的序号相匹配，则会在processFilteredPacket里处理一些若干个连续的Packet：

Reason oret = REASON_SUCCESS;
PacketPtr pCurrPacket = p;
PacketPtr pNextPacket = NULL;

// push this packet chain (frequently one) through processOrderedPacket

// NOTE: We check isCondemned on the channel and not isDead. If a channel
// has isDestroyed set to true but isCondemned false, we still want to
// process remaining messages. This can occur if there is a message that
// causes the entity to teleport. Any remaining messages are still
// processed and will likely be forwarded from the ghost entity to the
// recently teleported real entity.

// TODO: It would be nice to display a message if the channel is condemned
// but there are messages on it.

while (pCurrPacket &&
	((pChannel == NULL) || !pChannel->isCondemned()))
{
	// processOrderedPacket expects packets not to be chained, since
	// chaining is used for grouping fragments into bundles.  The packet
	// chain we've set up doesn't have anything to do with bundles, so we
	// break the packet linkage before passing the packets into
	// processOrderedPacket.  This can mean that packets that aren't the one
	// just received drop their last reference, hence the need for
	// pCurrPacket and pNextPacket.
	pNextPacket = pCurrPacket->next();
	pCurrPacket->chain( NULL );

	// Make sure they are actually packets with consecutive sequences.
	MF_ASSERT( pNextPacket.get() == NULL || 
		seqMask( pCurrPacket->seq() + 1 ) == 
			pNextPacket->seq() );

	// At this point, the only footers left on the packet should be the
	// request and fragment footers.
	Reason ret = this->processOrderedPacket( addr, pCurrPacket.get(),
			pChannel.get(), pStatsHelper );

	if (oret == REASON_SUCCESS)
	{
		oret = ret;
	}

	pCurrPacket = pNextPacket;
}

这里使用while来消耗掉已经接收到的多个连续的Packet，每个Packet都会经过processOrderedPacket的处理。由于单Bundle可能是由多个packet组合而成的，同时一个Packet里可能会有多个Bundle的数据，所以这里使用一个比Packet更小粒度的概念FragmentedBundle来代表单一Bundle里的多个连续区块, 其概念上类似于绑定了一个shared_ptr<Bundle>的string_view。然后完整的Bundle就由一个FragmentedBundle的单链表来组成。当单一bundle的所有packet都到达之后，这里会使用UDPBundleProcessor::dispatchMessages来将当前完整的bundle进行消息回调:

/**
 * Process a packet after any ordering guaranteed by reliable channels
 * has been imposed (further ordering guaranteed by fragmented bundles
 * is still to be imposed)
 */
Reason PacketReceiver::processOrderedPacket( const Address & addr, Packet * p,
	UDPChannel * pChannel, ProcessSocketStatsHelper * pStatsHelper )
{
	// 省略很多代码
	// We have a complete packet chain.  We can drop the reference in pChain now
	// since the Bundle owns it.
	UDPBundleProcessor outputBundle( p );
	pChain = NULL;

	Reason reason = outputBundle.dispatchMessages(
			networkInterface_.interfaceTable(),
			addr,
			pChannel,
			networkInterface_,
			pStatsHelper );

	if (reason == REASON_CORRUPTED_PACKET)
	{
		RETURN_FOR_CORRUPTED_PACKET();
	}
}

由于一个bundle里可能会有多个Message，所以在这个dispatchMessages里会使用一个循环来处理内部存储的所有消息:

/**
 *	This method is responsible for dispatching the messages on this bundle to
 *	the appropriate handlers.
 *
 *	@param interfaceTable 	The interface table.
 *	@param addr 			The source address of the bundle.
 *	@param pChannel 		The channel.
 *	@param networkInterface The network interface.
 *	@param pStatsHelper 	The socket receive statistics.
 *
 *	@return 				REASON_SUCCESS on success, otherwise an appropriate
 *							Mercury::Reason describing the error.
 */
Reason UDPBundleProcessor::dispatchMessages( InterfaceTable & interfaceTable,
		const Address & addr, UDPChannel * pChannel,
		NetworkInterface & networkInterface, 
		ProcessSocketStatsHelper * pStatsHelper ) const
{
#	define SOURCE_STR (pChannel ? pChannel->c_str() : addr.c_str())
	bool breakLoop = pChannel ? pChannel->isDead() : false;
	Reason ret = REASON_SUCCESS;

	// NOTE: The channel may be destroyed while processing the messages so we
	// need to hold a local reference to keep pChannel valid. 
	ChannelPtr pChannelHolder = pChannel;
	MessageFilterPtr pMessageFilter =
		pChannel ? pChannel->pMessageFilter() : NULL;

	// now we simply iterate over the messages in that bundle
	iterator iter	= this->begin();
	iterator end	= this->end();

	interfaceTable.onBundleStarted( pChannel );

	while (iter != end && !breakLoop)
	{
		// find out what this message looks like
		InterfaceElementWithStats & ie = interfaceTable[ iter.msgID() ];
		// 省略一些代码代码
		InterfaceElement updatedIE = ie;
		if (!updatedIE.updateLengthDetails( networkInterface, addr ))
		{
			ERROR_MSG( "UDPBundleProcessor::dispatchMessages( %s ): "
					"Discarding bundle after failure to update length "
					"details for message ID %hu\n",
				SOURCE_STR, (unsigned short int)iter.msgID() );
			ret = REASON_CORRUPTED_PACKET;
			break;
		}
		// get the details out of it
		UnpackedMessageHeader & header = iter.unpack( updatedIE );
		// 省略一些代码
	}
	// 省略很多代码
}

在处理单个消息的解析的时候，首先要做的就是获取当前消息的参数长度，这个逻辑在iter.unpack函数中:

/**
 *	This method unpacks the current message using the given
 *	interface element.
 *
 *	@param ie	InterfaceElement for the current message.
 *
 *	@return		Header describing the current message.
 */
UnpackedMessageHeader & UDPBundleProcessor::iterator::unpack( 
		const InterfaceElement & ie )
{
	uint16	msgBeg = offset_;

	MF_ASSERT( !isUnpacked_ );

	bool isRequest = (nextRequestOffset_ == offset_);

	updatedIE_ = ie;
	// read the standard header
	if (int(offset_) + updatedIE_.headerSize() > int(bodyEndOffset_))
	{
		ERROR_MSG( "UDPBundleProcessor::iterator::unpack( %s ): "
				"Not enough data on stream at %hu for header "
				"(%d bytes, needed %d)\n",
			updatedIE_.name(), offset_, int(bodyEndOffset_) - int(offset_),
			updatedIE_.headerSize() );

		goto errorNoRevert;
	}

	curHeader_.identifier = this->msgID();
	curHeader_.length =
		updatedIE_.expandLength( cursor_->data() + msgBeg, cursor_, isRequest );
	// 省略很多代码
}

在UDPBundleProcessor::iterator::unpack我们终于见到了之前介绍过的compressLength的逆过程expandLength，在执行完成expandlength之后，当前消息的总长度就存储在curHeader_.length里。有了总长度之后，就可以方便的知道后续哪些packet里有当前message的消息，当所有packet都收集完成了之后，下面的这个函数就会将分散在各个packet里的参数数据进行合并，成为一个单一的连续buffer:

/**
 * 	This method returns the data for the message that the iterator
 * 	is currently pointing to.
 *
 * 	@return 	Pointer to message data.
 */
const char * UDPBundleProcessor::iterator::data()
{
	// does this message go off the end of the packet?
	if (dataOffset_ + dataLength_ <= bodyEndOffset_)
	{
		// no, ok, we're safe
		return cursor_->data() + dataOffset_;
	}

	// is there another packet? assert that there is because 'unpack' would have
	// flagged an error if the next packet was required but missing
	MF_ASSERT( cursor_->next() != NULL );
	if (cursor_->next() == NULL) return NULL;
	// also assert that data does not start mid-way into the next packet
	MF_ASSERT( dataOffset_ <= bodyEndOffset_ );	// (also implied by 'unpack')

	// is the entirety of the message data on the next packet?
	if (dataOffset_ == bodyEndOffset_ &&
		Packet::HEADER_SIZE + dataLength_ <= cursor_->next()->msgEndOffset())
	{
		// yes, easy then
		return cursor_->next()->body();
	}

	// ok, it's half here and half there, time to make a temporary buffer.
	// note that a better idea might be to return a stream from this function.

	if (dataBuffer_ != NULL)
	{
		// Already created a buffer for it.  
		return dataBuffer_;
	}

	// Buffer is destroyed in operator++() and in ~iterator().
	dataBuffer_ = new char[dataLength_];
	Packet *thisPack = cursor_;
	uint16 thisOff = dataOffset_;
	uint16 thisLen;
	for (int len = 0; len < dataLength_; len += thisLen)
	{
		if (thisPack == NULL)
		{
			DEBUG_MSG( "UDPBundleProcessor::iterator::data: "
				"Run out of packets after %d of %d bytes put in temp\n",
				len, dataLength_ );
			return NULL;
		}
		thisLen = thisPack->msgEndOffset() - thisOff;
		if (thisLen > dataLength_ - len) thisLen = dataLength_ - len;
		memcpy( dataBuffer_ + len, thisPack->data() + thisOff, thisLen );
		thisPack = thisPack->next();
		thisOff = Packet::HEADER_SIZE;
	}
	return dataBuffer_;
}

这里的dataBuffer_就是最终的参数数据的连续buffer，通过这个连续buffer可以构造BinaryIStream，通过operator>>的方式来解析出原来的所有参数。

Channel与可靠UDP

在本书的开头章节中我们介绍过UDP不是一个业务层能直接使用的协议，因为UDP数据包在发送、传输、接收过程中有各种理由能导致数据丢失和乱序。为了能让业务层能够相信网络功能与TCP一样能够提供可靠有序的消息消息收发，网络框架这一层必须提供一个可靠UDP的实现，来抹平底层的网络协议差异，典型样例便是KCP协议。在BigWorld中，业务通信主要使用的都是UDP，因此其网络层提供了一个自己实现的可靠UDP， 叫做UDPChannel:

/**
 *	Channels are used for regular communication channels between two address.
 *
 *	@note Any time you call 'bundle' you may get a different bundle to the one
 *	you got last time, because the Channel decided that the bundle was full
 *	enough to send. This does not occur on high latency channels (or else
 *	tracking numbers would get very confusing).
 *
 *	@note If you use more than one Channel on the same address, they share the
 *	same bundle. This means that:
 *
 *	@li Messages (and message sequences where used) must be complete between
 *		calls to 'bundle' (necessary due to note above anyway)
 *
 *	@li Each channel must say send before the bundle is actually sent.
 *
 *	@li Bundle tracking does not work with multiple channels; only the last
 *		Channel to call 'send' receives a non-zero tracking number (or possibly
 *		none if deleting a Channel causes it to be sent), and only the first
 *		Channel on that address receives the 'bundleLost' call.
 *
 * 	@ingroup mercury
 */
class UDPChannel : public Channel
{
	// 省略很多代码
};

UDPChannel的父类Channel就是一个逻辑上的数据收发通道，这个Channel类负责提供基于Bundle的发送接收的虚接口。Channel有一个另外的子类TCPChannel来处理基于TCP的数据收发，由于TCP已经是一个可靠协议，所以TCPChannel的逻辑并不多，直接把Bundle加上长度字段即可塞入到发送队列中。而UDPChannel则需要在底层将一个Bundle的数据拆分为多个Packet，并依次发送到消息的对端，因为如果单个消息包太大超过以太网的常见MTU的话，会导致拆包，从而增大了丢包和乱序的风险。UDPChannel对端接收数据的时候也需要使用UDPChannel，只有两端都是UDPChannel的时候才能使用框架提供的能力来实现可靠传输。接下来我们来大致的分析一下这个UDPChannel是如何实现可靠传输的。

在UDP上做可靠传输，首先需要为发出的每个包分配一个连续自增的编号，在Packet上也提供了这个字段:

/// Sequence number, or Channel::SEQ_NULL if not set
SeqNum		seq_;

这个字段在Packet刚被创建的时候是默认为0的，只有在PacketSender::send执行的时候才会通过UDPBundle::preparePackets来执行填充:

/**
 * 	This method sends a bundle to the given address.
 *
 * 	Note: any pointers you have into the packet may become invalid after this
 * 	call (and whenever a channel calls this too).
 *
 * 	@param address	The address to send to.
 * 	@param bundle	The bundle to send
 *	@param pChannel	The Channel that is sending the bundle.
 *				(even if the bundle is not sent reliably, it is still passed
 *				through the filter associated with the channel).
 */
void PacketSender::send( const Address & address,
		UDPBundle & bundle, UDPChannel * pChannel )
{
	MF_ASSERT( address != Address::NONE );
	MF_ASSERT( !pChannel || pChannel->addr() == address );

	MF_ASSERT( !bundle.pChannel() || (bundle.pChannel() == pChannel) );

#if ENABLE_WATCHERS
	sendingStats_.mercuryTimer().start();
#endif // ENABLE_WATCHERS

	if (!bundle.isFinalised())
	{
		// Handle bundles sent off-channel that won't have been finalised by
		// their channels yet.
		bundle.finalise();
		bundle.addReplyOrdersTo( &requestManager_, pChannel );
	}

	// fill in all the footers that are left to us
	Packet * pFirstOverflowPacket = bundle.preparePackets( pChannel,
				seqNumAllocator_, sendingStats_, shouldUseChecksums_ );

	// 省略后续的往socket投递的逻辑
}

这个函数有点长，目前我们先关注关于序列号分配部分的逻辑：

/**
 *	This method prepares packets this bundle for sending.
 *
 *	@param pChannel			The channel, or NULL for off-channel sending.
 *	@param seqNumAllocator 	The network interface's sequence number allocator,
 *							used for off-channel sending.
 *	@param sendingStats 	The sending stats to update.
 */
Packet * UDPBundle::preparePackets( UDPChannel * pChannel,
		SeqNumAllocator & seqNumAllocator,
		SendingStats & sendingStats,
		bool shouldUseChecksums )
{
	// fill in all the footers that are left to us
	Packet * pFirstOverflowPacket = NULL;

	int	numPackets = this->numDataUnits();
	SeqNum firstSeq = 0;
	SeqNum lastSeq = 0;

	// Write footers for each packet.
	for (Packet * pPacket = this->pFirstPacket();
			pPacket;
			pPacket = pPacket->next())
	{
		MF_ASSERT( pPacket->msgEndOffset() >= Packet::HEADER_SIZE );

		// 省略很多代码

		this->writeFlags( pPacket );

		if (pChannel)
		{
			pChannel->writeFlags( pPacket );
		}

		if ((pChannel && pChannel->isExternal()) ||  
			pPacket->hasFlags( Packet::FLAG_IS_RELIABLE ) || 
			pPacket->hasFlags( Packet::FLAG_IS_FRAGMENT )) 
		{ 
			pPacket->reserveFooter( sizeof( SeqNum ) ); 
			pPacket->enableFlags( Packet::FLAG_HAS_SEQUENCE_NUMBER ); 
		} 

		// At this point, pPacket->back() is positioned just after the message
		// data, so we advance it to the end of where the footers end, then
		// write backwards towards the message data. We check that we finish
		// up back at the message data as a sanity check.
		const int msgEndOffset = pPacket->msgEndOffset();
		pPacket->grow( pPacket->footerSize() );

		
		// 省略很多代码
		// Add the sequence number
		if (pPacket->hasFlags( Packet::FLAG_HAS_SEQUENCE_NUMBER ))
		{
			// If we're sending reliable traffic on a channel, use the
			// channel's sequence numbers.  Otherwise use the nub's.
			pPacket->seq() =
				(pChannel && pPacket->hasFlags( Packet::FLAG_IS_RELIABLE )) ?
					pChannel->useNextSequenceID() :
					seqNumAllocator.getNext();
			
			pPacket->packFooter( pPacket->seq() );

			if (pPacket == pFirstPacket_)
			{
				firstSeq = pPacket->seq();
				lastSeq = pPacket->seq() + numPackets - 1;
			}
		}

		// 省略很多代码
	}

	return pFirstOverflowPacket;
}

这个preparePackets内部用循环去处理当前Bundle里的每一个Packet。如果当前Bundle是一个需要执行可靠收发的Bundle，则内部的所有Packet都会有Packet::FLAG_IS_RELIABLE这个可靠性的flag，此时会通过reserveFooter在当前Packet的末尾预留四个字节大小的长度来等待后续的序列号填充，并顺带的会开启Packet::FLAG_HAS_SEQUENCE_NUMBER这个需要携带序列号的标记位。这里的reserveFooter并不会分配额外的动态内存，而是累计到当前footerSize_上，作为当前packet的footer部分的总大小：

void reserveFooter( int nBytes ) { footerSize_ += nBytes; }
int footerSize() const		{ return footerSize_; }

当所有的footer都被统计了之后，会使用Packet::grow来在原有的数据后面进行内存边界扩张:

void grow( int nBytes )				{ msgEndOffset_ += nBytes; }

这里grow的时候不需要担心msgEndOffset_这个偏移量会超过内部data_数组的最大容量，因为Packet会预先留一个最大可能的RESERVED_FOOTER_SIZE字节不被填参的时候占用，统计当前Packet可用内存的时候会预先对这块区域做扣除:

/// The amount of space that is reserved for fixed-length footers on a
/// packet.  This is done so that the bundle logic can always assume that
/// these footers will fit and not have to worry about pre-allocating them.
/// This is currently 27 bytes, roughly 1.5% of the capacity of a packet, so
/// there's not too much wastage.
static const int RESERVED_FOOTER_SIZE =
	sizeof( Offset ) + // FLAG_HAS_REQUESTS
	sizeof( AckCount ) + // FLAG_HAS_ACKS
	sizeof( SeqNum ) + // FLAG_HAS_SEQUENCE_NUMBER
	sizeof( SeqNum ) * 2 + // FLAG_IS_FRAGMENT
	sizeof( ChannelID ) + sizeof( ChannelVersion ) + // FLAG_INDEXED_CHANNEL
	sizeof( Checksum ); // FLAG_HAS_CHECKSUM

int freeSpace() const
{
	return MAX_SIZE -
		RESERVED_FOOTER_SIZE -
		msgEndOffset_ -
		footerSize_ -
		extraFilterSize_;
}

然后使用的时候，使用专门的接口packFooter，这个接口会将传入数据的字节优先从data_的末尾进行填充，所以当所有的footer都填充完成之后，msgEndOffset_指向的就是真正数据的最大偏移，在这个msgEndOffset_之后的footerSize_字节对应的数据全都是footer对应的数据：

/**
	*  This method writes a footer to the back of this packet.  It should only
	*  be called from NetworkInterface::send() and assumes that size_ has been
	*  artificially increased so that it points to the end of the footers, the
	*  idea being that we work back towards the real body end.
	*/
template <class TYPE>
void packFooter( TYPE value )
{
	msgEndOffset_ -= sizeof( TYPE );

	switch( sizeof( TYPE ) )
	{
		case sizeof( uint8 ):
			*(TYPE*)this->back() = value; break;

		case sizeof( uint16 ):
			*(TYPE*)this->back() = BW_HTONS( value ); break;

		case sizeof( uint32 ):
			*(TYPE*)this->back() = BW_HTONL( value ); break;

		default:
			CRITICAL_MSG( "Footers of size %" PRIzu " aren't supported",
				sizeof( TYPE ) );
	}
}

preparePackets后面的逻辑里发现当前packet拥有Packet::FLAG_HAS_SEQUENCE_NUMBER这个标记位的时候，就会生成一个递增序列号并使用packFooter将这个序列号填充到之前预留的位置上。这里生成递增序列号有两个途径，正常情况下我们的RPC对应的packet都是Packet::FLAG_IS_RELIABLE 的，因此会走UDPChannel::useNextSequenceID:

/**
 * 	This method returns the next sequence ID, and then increments it.
 *
 * 	@return The next sequence ID.
 */
SeqNum UDPChannel::useNextSequenceID()
{
	SeqNum	retSeq = largeOutSeqAt_.getNext();

	if (this->isInternal())
	{
		int usage = this->sendWindowUsage();
		int & threshold = this->sendWindowWarnThreshold();

		if (usage > threshold)
		{
			WARNING_MSG( "UDPChannel::useNextSequenceID( %s ): "
							"Send window backlog is now %d packets, "
							"exceeded previous max of %d, "
							"critical size is %u\n",
						this->c_str(), usage, threshold, windowSize_ );

			threshold = usage;
		}

		if (this->isIndexed() &&
				(s_pSendWindowCallback_ != NULL) &&
				(usage > s_sendWindowCallbackThreshold_))
		{
			(*s_pSendWindowCallback_)( *this );
		}
	}

	return retSeq;
}

这里的largeOutSeqAt_是UDPChannel内的一个成员变量，作为递增序列号发生器来使用。在这里生成下一个递增序列号的时候，会顺带的检查一下现在的发送窗口里有多少个Packet还没有发送出去，如果数值太大则会有一个警告，同时会触发一个s_pSendWindowCallback_的回调。

在preparePackets填充好每个Packet的递增序列号之后，会使用addResendTimer对每个需要可靠发送的包注册为可以消息重传的Packet:

// set up the reliable machinery
if (pPacket->hasFlags( Packet::FLAG_IS_RELIABLE ))
{
	if (pChannel)
	{
		const ReliableOrder *roBeg, *roEnd;

		if (pChannel->isInternal())
		{
			roBeg = roEnd = NULL;
		}
		else
		{
			this->reliableOrders( pPacket, roBeg, roEnd );
		}

		if (!pChannel->addResendTimer( pPacket->seq(), pPacket, 
				roBeg, roEnd ))
		{
			if (pFirstOverflowPacket == NULL)
			{
				pFirstOverflowPacket = pPacket;
			}
			// return REASON_WINDOW_OVERFLOW;
		}
		else
		{
			MF_ASSERT( pFirstOverflowPacket == NULL );
		}
	}
}

在这个addResendTimer中，会为每个要发送的Packet构造一个UnackedPacket，同时记录一下现在的时间，并将这个UnackedPacket放到当前UDPChannel的unackedPackets_数组里去，

/**
 *	This method records a packet that may need to be resent later if it is not
 *	acknowledged. It is called when a packet is sent on our behalf.
 *
 *	@return false if the window size was exceeded.
 */
bool UDPChannel::addResendTimer( SeqNum seq, Packet * p,
		const ReliableOrder * roBeg, const ReliableOrder * roEnd )
{
	MF_ASSERT( (oldestUnackedSeq_ == SEQ_NULL) ||
			unackedPackets_[ oldestUnackedSeq_ ] );
	MF_ASSERT( seq == p->seq() );

	UnackedPacket * pUnackedPacket = new UnackedPacket( p );

	// If this channel has no unacked packets, record this as the oldest.
	if (oldestUnackedSeq_ == SEQ_NULL)
	{
		oldestUnackedSeq_ = seq;
	}

	// Fill it in
	pUnackedPacket->lastSentAtOutSeq_ = seq;

	uint64 now = timestamp();
	pUnackedPacket->lastSentTime_ = now;
	lastReliableSendTime_ = now;

	pUnackedPacket->wasResent_ = false;

	if (roBeg != roEnd)
	{
		pUnackedPacket->reliableOrders_.assign( roBeg, roEnd );
	}

	// Grow the unackedPackets_ array, if necessary.
	if (seqMask( seq - oldestUnackedSeq_ + 1 ) > unackedPackets_.size())
	{
		unackedPackets_.doubleSize( oldestUnackedSeq_ );

		if (this->networkInterface().isVerbose())
		{
			INFO_MSG( "UDPChannel::addResendTimer( %s ): "
					"Doubled send buffer size to %u\n",
				this->c_str(),
				unackedPackets_.size() );
		}
	}

	MF_ASSERT( unackedPackets_[ seq ] == NULL );
	unackedPackets_[ seq ] = pUnackedPacket;

	MF_ASSERT( (oldestUnackedSeq_ == SEQ_NULL) ||
			unackedPackets_[ oldestUnackedSeq_ ] );

	if (seqMask( largeOutSeqAt_ - oldestUnackedSeq_ ) >= windowSize_)
	{
		// Make sure that we at least send occasionally.
		UnackedPacket * pPrevUnackedPacket =
			unackedPackets_[ seqMask( smallOutSeqAt_ - 1 ) ];

		if ((pPrevUnackedPacket == NULL) ||
			(now - pPrevUnackedPacket->lastSentTime_ >
				minInactivityResendDelay_))
		{
			this->sendUnacked( *unackedPackets_[ smallOutSeqAt_ ] );
			smallOutSeqAt_ = seqMask( smallOutSeqAt_ + 1 );
		}

		this->checkOverflowErrors();
		//We shouldn't send now
		return false;
	}
	else
	{
		//We should send now
		smallOutSeqAt_ = largeOutSeqAt_;
		return true;
	}
}

这个函数的后半部分差不多实现了一个类似于TCP里的滑动窗口的概念，oldestUnackedSeq_是当前最小的没有收到对端ACK的Packet的序号，而largeOutSeqAt_代表的是当前已发送出去的Packet的最大序号， smallOutSeqAt_代表的是等待发送的最小Packet序号:

uint32			windowSize_;

/// Generally, the sequence number of the next packet to be sent.
SeqNum			smallOutSeqAt_; // This does not include packets in 
								// overflowPackets_

SeqNumAllocator	largeOutSeqAt_; // This does include packets in 
								// overflowPackets_

/// The sequence number of the oldest unacked packet on this channel.
SeqNum			oldestUnackedSeq_;

/// The last time a reliable packet was sent (for the first time) on this
/// channel, as a timestamp.
uint64			lastReliableSendTime_;

如果largeOutSeqAt_与oldestUnackedSeq_两者的差值大于等于指定的windowSize_，则代表目前已发送但是没有确认接受的包已经超过了阈值。此时会认为当前包要暂存下来，不能直接发出，等待窗口有余量的时候再发，避免信道拥堵时的进一步加重，所以这里会返回false。在返回之前会判断一下近期一段时间minInactivityResendDelay_内是否有新的包发从出去，如果没有的话则强制发出下一个等待处理的包，并更新smallOutSeqAt_++。这样做的目的是强制推进一下滑动窗口，通知对端当前已经收到的ACK的最大值，类似于一种心跳机制。

如果当前已发送但没有ack的包的数量不是很多，则认为当前Packet可以发出，函数返回true。返回之前会更新smallOutSeqAt_为当前已发送出去的Packet的最大序号largeOutSeqAt_。网络状况好的话当前没有未ACK的Packet，此时largeOutSeqAt_就是当前新Packet的序列号。

发送端除了处理滑动窗口之外，还需要处理已发送包的ACK, 这部分逻辑是通过函数handleAck来处理的:

/**
 *	This method removes a packet from the collection of packets that have been
 *	sent but not acknowledged. It is called when an acknowledgement to a packet
 *	on this channel is received.
 *
 *  Returns false on error, true otherwise.
 */
bool UDPChannel::handleAck( SeqNum seq )
{
	MF_ASSERT( (oldestUnackedSeq_ == SEQ_NULL) ||
			unackedPackets_[ oldestUnackedSeq_ ] );

	// Make sure the sequence number is valid
	// 忽略一些容错处理

	// now make sure there's actually a packet there
	UnackedPacket * pUnackedPacket = unackedPackets_[ seq ];
	if (pUnackedPacket == NULL)
	{
		return true;
	}

	// Update the average RTT for this channel, if this packet hadn't already
	// been resent.
	// 忽略一些计算RTT的代码

	// 忽略一些无关代码

	// If we released the oldest unacked packet, figure out the new one
	if (seq == oldestUnackedSeq_)
	{
		oldestUnackedSeq_ = SEQ_NULL;
		for (uint i = seqMask( seq+1 );
			 i != largeOutSeqAt_;
			 i = seqMask( i+1 ))
		{
			if (unackedPackets_[ i ])
			{
				oldestUnackedSeq_ = i;
				break;
			}
		}
	}

	// If the incoming seq is after the last ack, then it is the new last ack
	if (seqLessThan( highestAck_, seq ))
	{
		highestAck_ = seq;
	}

	// Now we can release the unacked packet
	bw_safe_delete( pUnackedPacket );
	unackedPackets_[ seq ] = NULL;

	MF_ASSERT( oldestUnackedSeq_ == SEQ_NULL ||
			unackedPackets_[ oldestUnackedSeq_ ] );

	while (seqMask(smallOutSeqAt_ - oldestUnackedSeq_) < windowSize_ &&
		   unackedPackets_[ smallOutSeqAt_ ])
	{
		this->sendUnacked( *unackedPackets_[ smallOutSeqAt_ ] );
		smallOutSeqAt_ = seqMask( smallOutSeqAt_ + 1 );
	}

	return true;
}

这个函数的逻辑比较简单， 可以分为这四个部分：

1. 如果当前被ACK的包就是之前没有被ACK的最小序号oldestUnackedSeq_，则通过遍历后续所有已发送包里获取其中第一个未ACK的序号执行oldestUnackedSeq_的更新。这里不直接进行++操作是因为handleAck可能会被乱序执行，即可能先收到5的ACK ，再收到4的ACK。
2. 更新highestAck_为当前已接受到的最大ACK，注意这里并不代表highestAck_之前的所有包都已经被ACK了
3. 通过bw_safe_delete来释放之前存储的Packet，因为已经被ACK的包不可能再被重传，所以没必要再保留备份
4. 如果当前的滑动窗口没有被塞满，则可以将下一个等待发送的包smallOutSeqAt_发送出去，同时更新smallOutSeqAt_++

如果对端通过某种机制通知到当前的UDPChannel所有指定序列号endSeq之前的包都已经被顺序接收了，则UDPChannel::handleCumulativeAck会被调用。在这个函数里会获取最小的没有被ack的包序号oldestUnackedSeq_，然后遍历[oldestUnackedSeq_, endSeq)区间内的所有包来调用handleAck：

/**
 *	This method handles a cumulative ACK. This indicates that all packets
 *	BEFORE a sequence number have been received by the remote end.
 *
 *  @return False on error, true otherwise.
 */
bool UDPChannel::handleCumulativeAck( SeqNum endSeq )
{
	// Make sure the sequence number is valid
	// 忽略一些容错代码

	if (!this->hasUnackedPackets())
	{
		return true;
	}

	// Check that the ACK is not in the future.
	// Note: endSeq is first seqNum after what's been received.
	// 忽略一些容错代码

	SeqNum seq = oldestUnackedSeq_;

	// Note: Up to but not including endSeq
	while (seqLessThan( seq, endSeq ))
	{
		this->handleAck( seq );
		seq = seqMask( seq + 1 );
	}

	return true;
}

如果一些包长时间没有被ACK，则需要主动的对这些包进行重传，这部分的逻辑在UDPChannel::checkResendTimers函数里，每次发送一个包之前都会被调用:

/**
 *	This method sends the given bundle on this channel. If no bundle is
 *	supplied, the channel's own bundle will be sent.
 *
 *	@param pBundle 	The bundle to send, or NULL to send the channel's own
 *					bundle.
 */
void Channel::send( Bundle * pBundle /* = NULL */ )
{
	// 省略很多代码

	this->doPreFinaliseBundle( *pBundle );

	// 省略很多代码
}
/*
 *	Override from Channel.
 */
void UDPChannel::doPreFinaliseBundle( Bundle & bundle )
{
	// Tack on piggybacks.

	UDPBundle & udpBundle = static_cast< UDPBundle & >( bundle );
	this->checkResendTimers( udpBundle );
}


/**
 *	This method resends any unacked packets as appropriate. This can be because
 *	of time since last sent, receiving later acks before earlier ones.
 */
void UDPChannel::checkResendTimers( UDPBundle & bundle )
{
	// There are no un-acked packets
	if (oldestUnackedSeq_ == SEQ_NULL)
	{
		return;
	}

	// Don't do anything if the remote process has failed
	// 忽略对端下线的情况

	// If we have unacked packets that are getting a bit old, then resend the
	// ones that are older than we'd like.  Anything that has taken more than
	// twice the RTT on the channel to come back is considered to be too old.
	uint64 now = timestamp();
	uint64 resendPeriod =
		std::max( roundTripTime_*2, minInactivityResendDelay_ );
	uint64 lastReliableSendTime = this->lastReliableSendOrResendTime();

	const bool isIrregular = !this->isRemoteRegular();
	const SeqNum endSeq = isIrregular ? smallOutSeqAt_ : highestAck_;
	const bool isDebugVerbose = this->networkInterface().isDebugVerbose();

	int numResends = 0;

	// TODO: 8 is a magic number and would be nice to be more scientific.
	// The idea is to throttle the resends a little in extreme situations. We
	// want to send enough so that no (or not too many) packets are lost but
	// still be able to send more when the RTT is large.
	const int MAX_RESENDS = windowSize_/8;

	for (SeqNum seq = oldestUnackedSeq_;
		seqLessThan( seq, endSeq ) && numResends < MAX_RESENDS;
		seq = seqMask( seq + 1 ))
	{
		UnackedPacket * pUnacked = unackedPackets_[ seq ];

		// Send if the packet is old, or we have a later ack
		if (pUnacked != NULL)
		{
			const bool hasNewerAck =
				 seqLessThan( pUnacked->lastSentAtOutSeq_, highestAck_);

			const bool shouldResend = hasNewerAck ||
				(isIrregular && (now - pUnacked->lastSentTime_ > resendPeriod));

			const SeqNum prevLastSentAtOutSeq = pUnacked->lastSentAtOutSeq_;
			const uint64 prevLastSentTime = pUnacked->lastSentTime_;

			if (shouldResend)
			{
				bool piggybacked = this->resend( seq, bundle );
				++numResends;
				// 忽略一些警告代码
			}
		}
	}
}

在这个函数里会首先计算出当前能够容忍的最长未确认时间resendPeriod，然后遍历当前所有的已发出但未确认的UnackedPacket，如果这些UnackedPacket的发送后时长大于最长未确认时间resendPeriod，则会调用resend将这个Packet重新发送:

/**
 *  Resends an un-acked packet by the most sensible method available.
 *
 *  @return Returns true if the packet is no longer un-acked.
 */
bool UDPChannel::resend( SeqNum seq, UDPBundle & bundle )
{
	++numPacketsResent_;

	UnackedPacket & unacked = *unackedPackets_[ seq ];

	// If possible, piggypack this packet onto the next outgoing bundle
	if (this->isExternal() &&
		!unacked.pPacket_->hasFlags( Packet::FLAG_IS_FRAGMENT ) &&
		(unackedPackets_[ smallOutSeqAt_ ] == NULL)) // Not going to overflow
	{
		if (bundle.piggyback(
				seq, unacked.reliableOrders_, unacked.pPacket_.get() ))
		{
			unacked.wasResent_ = true; // Don't count this for RTT calculations
			this->handleAck( seq );
			return true;
		}
	}

	// If there are any acks on this packet, then they will be resent too, but
	// it does no harm.
	this->sendUnacked( unacked );

	return false;
}

这个resend的最简单的处理逻辑就是执行sendUnacked来重发，记录一下这个包的重发时间戳和重发记录：

/**
 *  Resends an un-acked packet by the most sensible method available.
 */
void UDPChannel::sendUnacked( UnackedPacket & unacked )
{
	unacked.pPacket_->updateChannelVersion( version_, id_ );

	pNetworkInterface_->sendPacket( addr_, unacked.pPacket_.get(), this, 
		/* isResend: */ true );

	unacked.lastSentAtOutSeq_ = smallOutSeqAt_;
	unacked.wasResent_ = true;

	uint64 now = timestamp();
	unacked.lastSentTime_ = now;
	lastReliableResendTime_ = now;
}

如果当前是面向客户端的下行通道，且当前要重发的包包含了完整的一个bundle，同时滑动窗口里允许下一个包进行发送，那么这里会通过piggyback函数将当前Packet连接到当前要发送的bundle的Packet链表后面，并执行这个Packet的手动ACK。这样的快速路径相当于将这个要重传的Packet与当前要发送的bundle进行合并了。

至此UDPChannel的发送端实现了序号分配、滑动窗口、接收确认、超时重传等流控和可靠机制，基本等价实现了一个TCP的发送端。接下来我们再来看看UDPChannel的接收端是如何收取Packet并返回ACK的，相关代码在之前我们已经介绍过的PacketReceiver::processFilteredPacket函数里，不过之前只介绍了接收端如何从Packet里解包Bundle的代码，并没有关注ACK的部分。现在我们再来看看这个函数是怎么发出ACK的：

/**
 *	This function has to be very robust, if we intend to use this transport over
 *	the big bad internet. We basically have to assume it'll be complete garbage.
 */
Reason PacketReceiver::processFilteredPacket( const Address & addr,
		Packet * p, ProcessSocketStatsHelper * pStatsHelper )
{
	// 省略很多代码
		// now do something if it's reliable
	if (p->hasFlags( Packet::FLAG_IS_RELIABLE ))
	{
		// first make sure it has a sequence number, so we can address it
		// 省略一些容错代码

		// should we be looking in a channel
		if (pChannel)
		{
			UDPChannel::AddToReceiveWindowResult result =
				pChannel->addToReceiveWindow( p, addr, stats_ );

			if (!pChannel->isLocalRegular())
			{
				shouldSendChannel = true;
			}

			if (result != UDPChannel::PACKET_IS_NEXT_IN_WINDOW)
			{
				// The packet is not corrupted, and has either already been
				// received, or is too early and has been buffered. In either
				// case, we send the ACK immediately, as long as the channel is
				// established and is irregular.
				if (result != UDPChannel::PACKET_IS_CORRUPT)
				{
					if (pChannel->isEstablished() && shouldSendChannel)
					{
						UDPBundle emptyBundle;
						pChannel->send( &emptyBundle );
					}

					return REASON_SUCCESS;
				}

				// The packet has an invalid sequence number.
				else
				{
					RETURN_FOR_CORRUPTED_PACKET();
				}
			}
		}
	}
}

这里会根据addToReceiveWindow的返回之来做处理，如果返回值是PACKET_IS_NEXT_IN_WINDOW，则代表这个Packet刚好就是下一个被期望的序号，否则代表这个包是乱序接收的，此时会构建一个空UDPBundle来发送。这个空UDPBundle的作用就是立即ACK当前包,但是这里面的调用链其实比较晦涩，我们先跳过之前分析过的UDPChannel::send的调用链的具体内容，直接跳转到调用到的UDPBundle::preparePackets：

/**
 *	This method prepares packets this bundle for sending.
 *
 *	@param pChannel			The channel, or NULL for off-channel sending.
 *	@param seqNumAllocator 	The network interface's sequence number allocator,
 *							used for off-channel sending.
 *	@param sendingStats 	The sending stats to update.
 */
Packet * UDPBundle::preparePackets( UDPChannel * pChannel,
		SeqNumAllocator & seqNumAllocator,
		SendingStats & sendingStats,
		bool shouldUseChecksums )
{
	// fill in all the footers that are left to us
	Packet * pFirstOverflowPacket = NULL;

	int	numPackets = this->numDataUnits();
	SeqNum firstSeq = 0;
	SeqNum lastSeq = 0;

	// Write footers for each packet.
	for (Packet * pPacket = this->pFirstPacket();
			pPacket;
			pPacket = pPacket->next())
	{
		MF_ASSERT( pPacket->msgEndOffset() >= Packet::HEADER_SIZE );

		if (shouldUseChecksums)
		{
			// Reserve space for the checksum footer

			MF_ASSERT( !pPacket->hasFlags( Packet::FLAG_HAS_CHECKSUM ) );
			pPacket->reserveFooter( sizeof( Packet::Checksum ) );
			pPacket->enableFlags( Packet::FLAG_HAS_CHECKSUM );
		}

		this->writeFlags( pPacket );

		if (pChannel)
		{
			pChannel->writeFlags( pPacket );
		}
		// 省略很多代码
	}
	// 省略很多代码
}

这里的pChannel->writeFlags( pPacket )会在当前Packet还有足够剩余空间的时候将ACK信息携带进去:

/**
 *  This method will write the flags on a packet fitting for one that will ride
 *  on this channel. It will also reserve enough space for the footer.
 */
void UDPChannel::writeFlags( Packet * p )
{
	p->enableFlags( Packet::FLAG_ON_CHANNEL );

	// 忽略无关代码

	// Add a cumulative ACK. This indicates that all packets BEFORE a given seq
	// have been received.
	if (p->freeSpace() >= int( sizeof( SeqNum )))
	{
		p->enableFlags( Packet::FLAG_HAS_CUMULATIVE_ACK );
		p->reserveFooter( sizeof( SeqNum ) );

		Acks::iterator iter = acksToSend_.begin();

		while (iter != acksToSend_.end())
		{
			// Need to go through all due to wrap-around case.
			if (seqLessThan( *iter, inSeqAt_ ) )
			{
				acksToSend_.erase( iter++ );
			}
			else
			{
				++iter;
			}
		}
	}

	
	// 省略很多代码
}

如果当前Packet里剩余空间大于四个字节的话(对于空包来说这个显然成立)，这里会优先加入批量ACK的消息，在Packet的flags里加上Packet::FLAG_HAS_CUMULATIVE_ACK，然后在Footer里塞入下一个希望收到的连续包的编号inSeqAt_。这个inSeqAt_的更新是在UDPChannel::addToReceiveWindow函数里做的，如果新包的序号等于这个inSeqAt_，则对这个inSeqAt_不断递增，直到已经接受的乱序Packet里没有这个编号：

UDPChannel::AddToReceiveWindowResult UDPChannel::addToReceiveWindow( 
		Packet * p, const Address & srcAddr, PacketReceiverStats & stats )
{
	const SeqNum seq = p->seq();
	const bool isDebugVerbose = this->networkInterface().isDebugVerbose();
	// 省略很多的代码
	// check the good case first
	if (seq == inSeqAt_)
	{
		inSeqAt_ = seqMask( inSeqAt_ + 1 );

		Packet * pPrev = p;
		Packet * pBufferedPacket = bufferedReceives_[ inSeqAt_ ].get();

		// Attach as many buffered packets as possible to this one.
		while (pBufferedPacket != NULL)
		{
			// Link it to the prev packet then remove it from the buffer.
			pPrev->chain( pBufferedPacket );
			bufferedReceives_[ inSeqAt_ ] = NULL;
			--numBufferedReceives_;

			// Advance to the next buffered packet.
			pPrev = pBufferedPacket;
			inSeqAt_ = seqMask( inSeqAt_ + 1 );
			pBufferedPacket = bufferedReceives_[ inSeqAt_ ].get();
		}

		return PACKET_IS_NEXT_IN_WINDOW;
	}
	// 省略很多代码
}

当确定了要发送连续ACK之后，会从已接收但未ACK的Packet集合里删除所有序号比inSeqAt_小的，因为已经不再需要保留了。

除了这个连续ACK的加入之外，还需要在剩下的空余空间里塞入一些离散的ACK信息，这部分代码就在UDPChannel::writeFlags处理连续ACK代码的后面：

// Put on as many acks as we can.
if (!acksToSend_.empty() &&
	p->freeSpace() >= int(sizeof( Packet::AckCount ) + sizeof( SeqNum )))
{
	//Required to make GCC link this, something to do with templates
	const size_t MAX_ACKS = Packet::MAX_ACKS;
	p->enableFlags( Packet::FLAG_HAS_ACKS );
	p->reserveFooter( sizeof( Packet::AckCount ) );

	const size_t minSpace = p->freeSpace() / sizeof( SeqNum );
	const size_t minSpaceSize = std::min( minSpace, acksToSend_.size() );
	const size_t nAcks = std::min( minSpaceSize, MAX_ACKS );
	p->nAcks() = static_cast<Packet::AckCount>(nAcks);

	p->reserveFooter( sizeof( SeqNum ) * p->nAcks() );
}

这里会根据剩余空间的大小来算出当前Packet最多还能携带几个离散的ACK，存储在p->nAcks()里。等到真正执行写入的时候，会从acksToSend_里选择p->nAcks()数量的ACK序号塞入到footer里，同时从acksToSend_删除这些元素:

/**
 *  This method will write the appropriate flags on a packet to indicate that
 *  it is on this channel. It must be called after writeFlags.
 */
void UDPChannel::writeFooter( Packet * p )
{
	if (p->hasFlags( Packet::FLAG_INDEXED_CHANNEL ))
	{
		p->packFooter( p->channelID() );
		p->packFooter( p->channelVersion() );
	}

	if (p->hasFlags( Packet::FLAG_HAS_CUMULATIVE_ACK ))
	{
		p->packFooter( inSeqAt_ );
	}

	if (p->hasFlags( Packet::FLAG_HAS_ACKS ))
	{
		// Note: Technically we should start at inSeqAt_ since sequence numbers
		// wrap around but this is rare enough not to worry about (since it
		// still works but is less efficient).
		p->packFooter( (Packet::AckCount)p->nAcks() );
		uint acksAdded = 0;
		while (!acksToSend_.empty() && acksAdded < p->nAcks())
		{
			p->packFooter( *acksToSend_.begin() );
			acksToSend_.erase( acksToSend_.begin() );
			++acksAdded;
		}
	}
}

因为我们的ACK信息是附带在Packet里的，所以这个Packet丢失之后仍然会通过超时重传发送到对端。这里有一点递归的意思，可靠传输依赖于ACK，而ACK的发送又依赖于可靠传输。

当UDPChannel接收到一个携带了Packet::FLAG_HAS_CUMULATIVE_ACK或者Packet::FLAG_HAS_ACKS的Packet时，就会从footer里解析出这些ACK的包序号，执行之前我们介绍过的handleCumulativeAck和handleAck:

Reason PacketReceiver::processFilteredPacket( const Address & addr,
		Packet * p, ProcessSocketStatsHelper * pStatsHelper )
{
	// 省略很多代码
	f (p->hasFlags( Packet::FLAG_HAS_CUMULATIVE_ACK ))
	{
		if (!pChannel)
		{
			// 省略错误日志
		}

		SeqNum endSeq;

		if (!p->stripFooter( endSeq ))
		{
			// 省略错误日志
		}

		if (!pChannel->handleCumulativeAck( endSeq ))
		{
			RETURN_FOR_CORRUPTED_PACKET();
		}
	}
		// Strip and handle ACKs
	if (p->hasFlags( Packet::FLAG_HAS_ACKS ))
	{
		if (!p->stripFooter( p->nAcks() ))
		{
			// 省略错误日志
		}

		if (p->nAcks() == 0)
		{
			// 省略错误日志
		}

		// The total size of all the ACKs on this packet
		int ackSize = p->nAcks() * sizeof( SeqNum );

		// check that we have enough footers to account for all of the
		// acks the packet claims to have (thanks go to netease)
		if (p->bodySize() < ackSize)
		{
			// 省略错误日志
		}

		// For each ACK that we receive, we no longer need to store the
		// corresponding packet.
		if (pChannel)
		{
			for (uint i=0; i < p->nAcks(); i++)
			{
				SeqNum seq;

				if (!p->stripFooter( seq ))
				{
					// 省略错误日志
				}

				if (!pChannel->handleAck( seq ))
				{
					// 省略错误日志
				}
			}
		}
		// 省略很多代码
	}
	// 省略很多代码
}

到这里，整个ACK的发送和接收机制已经展示完毕，配合超时重传和滑动窗口，当前的UDPChannel基本模拟了可靠TCP。

Unreal Engine的网络通信

基础概念

Unreal Engine(后续简称为UE)能够使用同一份代码编译出游戏客户端和游戏专属服务器Dedicated Server，客户端与专属服务端之间的通信使用的是UDP。之所以选择UDP是因为UE从诞生起就跟FPS紧密连接，而FPS这类游戏对于网络延迟突然增大的容忍性是极其低的。UE在UDP的基础上也做了很多的封装，来实现客户端与服务端之间连接的可靠传输。这里我们就来介绍一下UE网络的相关实现细节。 UE网络通信中基础的概念主要包括如下五个：

1. NetDriver 网络处理的核心，负责管理 所有的物理数据连接，实际使用的是其子类IpNetDriver。里面封装了初始化客户端与服务器的连接，建立属性记录表，处理RPC函数，创建Socket，构建并管理当前Connection信息，接收数据包等等基本操作。服务器 NetDriver 维护一个 NetConnections 列表，每个连接代表一个连接的玩家客户端，负责复制 Actor 数据。客户端 NetDriver管理连接到服务器的单个连接

2. NetConnection 在服务器和客户端上，NetDriver 负责接收来自网络的数据包并将这些数据包传递给适当的 NetConnection(必要时建立新的 NetConnection)

3. Channel 数据通道 每一个通道只负责交换某一个特定类型特定实例的数据信息。UE中预先定义了如下几个通道:

   1. ControlChannel：客户端与服务器之间发送控制信息，主要是发送接收连接与断开的相关消息。在Connection中只会在初始化连接的时候创建一个该通道实例。
   2. ActorChannel：处理Actor本身相关信息的同步，包括自身的同步以及子组件，属性的同步，RPC调用等。每个Connection连接里的每个同步的Actor都对应着一个ActorChannel实例
   3. VoiceChannel：用于发送接收语音消息，在Connection中初始化连接的时候创建一个该通道实例

4. Packet 是在客户端服务端网络连接之间发送的数据，每次通道最后发出去的包都成为一个Packet, 数据内容由Packet元数据（如报头信息和确认 Ack）和 Bunches 组成。Packet是真正在UDP链路上发送的数据。

5. Bunch 是在客户端服务端网络连接的通道对之间发送的数据。当一个连接接收到一个数据包时，该数据包将被分解成单独的 Bunch，这些 Bunch 然后被传递到单独的通道以进一步处理。一个 Packet 可以不包含 Bunch、单个 Bunch 或者多个 Bunch。当一个 Bunch 太大时，在传输之前会把它切成许多小 Bunch，这些 Bunch 将被标记为 PartialInitial, Partial 或 PartialFinal。利用这些信息，在接收端重新组装 Bunch。然后Bunch又可以分为Reliable Bunch和Unreliable Bunch，Reliable Bunch会有丢包重传机制保证在接收端按照发送时的顺序接收，而Unreliable Bunch在丢包之后不会做任何处理。

消息发送

基本上所有的UE网络通信发包都会调用这个接口UChannel::SendBunch:

/** Send a bunch if it's not overflowed, and queue it if it's reliable. */
ENGINE_API virtual FPacketIdRange SendBunch(FOutBunch* Bunch, bool Merge);

例如在ReplicationGraph里执行单个Actor数据的向下同步时，会在数据都填充好之后使用这样的代码将这次数据从所属的Channel里向下发送:

int64 UReplicationGraph::ReplicateSingleActor_FastShared(AActor* Actor, FConnectionReplicationActorInfo& ConnectionData, 
	FGlobalActorReplicationInfo& GlobalActorInfo, UNetReplicationGraphConnection& ConnectionManager, const uint32 FrameNum)
{
	// 省略数据填充相关代码
	// Setup the connection specifics on the bunch before calling SendBunch
	OutBunch.ChName = ActorChannel->ChName;
	OutBunch.ChIndex = ActorChannel->ChIndex;
	OutBunch.Channel = ActorChannel;
	OutBunch.Next = nullptr;

	// SendIt
	{
		FGuardValue_Bitfield(ActorChannel->bHoldQueuedExportBunchesAndGUIDs, true);
		
		ActorChannel->SendBunch(&OutBunch, false);
	}
}

这里的FOutBunch的结构比较简单，继承自BitWriter，来承接基于bit的消息输入，此外还携带了很多传输控制信息，例如所属的Channel信息:

//
// A bunch of data to send.
//
class ENGINE_API FOutBunch : public FNetBitWriter
{
public:
	// Variables.
	FOutBunch *				Next;
	UChannel *				Channel;
	double					Time;
	int32					ChIndex;
	FName					ChName;
	int32					ChSequence;
	int32					PacketId;
	uint8					ReceivedAck:1;
	uint8					bOpen:1;
	uint8					bClose:1;
	uint8					bIsReplicationPaused:1;   // Replication on this channel is being paused by the server
	uint8					bReliable:1;
	uint8					bPartial:1;				// Not a complete bunch
	uint8					bPartialInitial:1;		// The first bunch of a partial bunch
	uint8					bPartialFinal:1;			// The final bunch of a partial bunch
	uint8					bHasPackageMapExports:1;	// This bunch has networkGUID name/id pairs
	uint8					bHasMustBeMappedGUIDs:1;	// This bunch has guids that must be mapped before we can process this bunch

	EChannelCloseReason		CloseReason;

	TArray< FNetworkGUID >	ExportNetGUIDs;			// List of GUIDs that went out on this bunch
	TArray< uint64 >		NetFieldExports;
};

这里的Time代表这个Bunch发送时的时间戳，bOpen/bClose是用来控制单个Channel的创建与销毁的，bReliable用来控制当前Bunch是否是需要可靠传输的消息, bPartial/bPartialInitial/bPartialFinal这三个字段负责控制Bunch的拆分。在执行SendBunch的时候，会首先使用IsBunchTooLarge判断当前要发送的Bunch是否大于最大单Bunch大小NetMaxConstructedPartialBunchSizeBytes=65536字节，数据量太大的话这个包会被直接丢弃。然后再判断当前Bunch能否与Connection里发送队列里的最后一个Bunch合并，这样合并多个包就可以合并为一个包，减少底层的UDP发送接口的调用次数:

// Fairly large number, and probably a bad idea to even have a bunch this size, but want to be safe for now and not throw out legitimate data
static int32 NetMaxConstructedPartialBunchSizeBytes = 1024 * 64;
static FAutoConsoleVariableRef CVarNetMaxConstructedPartialBunchSizeBytes(
	TEXT("net.MaxConstructedPartialBunchSizeBytes"),
	NetMaxConstructedPartialBunchSizeBytes,
	TEXT("The maximum size allowed for Partial Bunches.")
);

FPacketIdRange UChannel::SendBunch( FOutBunch* Bunch, bool Merge )
{
	if (IsBunchTooLarge(Connection, Bunch))
	{
		UE_LOG(LogNetPartialBunch, Error, TEXT("Attempted to send bunch exceeding max allowed size. BunchSize=%d, MaximumSize=%d"), Bunch->GetNumBytes(), NetMaxConstructedPartialBunchSizeBytes);
		Bunch->SetError();
		return FPacketIdRange(INDEX_NONE);
	}
	// 省略很多的代码
	// This is the max number of bits we can have in a single bunch
	const int64 MAX_SINGLE_BUNCH_SIZE_BITS  = Connection->GetMaxSingleBunchSizeBits();

	// Max bytes we'll put in a partial bunch
	const int64 MAX_SINGLE_BUNCH_SIZE_BYTES = MAX_SINGLE_BUNCH_SIZE_BITS / 8;

	// Max bits will put in a partial bunch (byte aligned, we dont want to deal with partial bytes in the partial bunches)
	const int64 MAX_PARTIAL_BUNCH_SIZE_BITS = MAX_SINGLE_BUNCH_SIZE_BYTES * 8;
	//-----------------------------------------------------
	// Contemplate merging.
	//-----------------------------------------------------
	int32 PreExistingBits = 0;
	FOutBunch* OutBunch = NULL;
	if
	(	Merge
	&&	Connection->LastOut.ChIndex == Bunch->ChIndex
	&&	Connection->LastOut.bReliable == Bunch->bReliable	// Don't merge bunches of different reliability, since for example a reliable RPC can cause a bunch with properties to become reliable, introducing unnecessary latency for the properties.
	&&	Connection->AllowMerge
	&&	Connection->LastEnd.GetNumBits()
	&&	Connection->LastEnd.GetNumBits()==Connection->SendBuffer.GetNumBits()
	&&	Connection->LastOut.GetNumBits() + Bunch->GetNumBits() <= MAX_SINGLE_BUNCH_SIZE_BITS )
	{
		// Merge.
		check(!Connection->LastOut.IsError());
		PreExistingBits = Connection->LastOut.GetNumBits();
		Connection->LastOut.SerializeBits( Bunch->GetData(), Bunch->GetNumBits() );
		Connection->LastOut.bOpen     |= Bunch->bOpen;
		Connection->LastOut.bClose    |= Bunch->bClose;

#if UE_NET_TRACE_ENABLED		
		SetTraceCollector(Connection->LastOut, GetTraceCollector(*Bunch));
		SetTraceCollector(*Bunch, nullptr);
#endif

		OutBunch                       = Connection->LastOutBunch;
		Bunch                          = &Connection->LastOut;
		check(!Bunch->IsError());
		Connection->PopLastStart();
		Connection->Driver->OutBunches--;
	}
	// 省略很多代码
}

执行Bunch合并的时候有一些过滤条件，要求归属于同一个Channel同时bReliable字段要一致，且当前Bunch被设置为允许合并且当前Connection也要设置为允许合并。最后还有一个最大Bunch大小的限制，要求合并之后的Bunch大小要小于MAX_SINGLE_BUNCH_SIZE_BITS,这个上限的设置目的是为了避免单个Bunch在使用UDP传输的时候被IP层的MTU限制导致链路层拆包。这里的MAX_SINGLE_BUNCH_SIZE_BITS的值是根据UNetConnection::GetMaxSingleBunchSizeBits()计算出来的，注意这里的单位是bit，而不是常见的字节:

/** The maximum number of bits allowed within a single bunch. */
FORCEINLINE int32 GetMaxSingleBunchSizeBits() const
{
	return (MaxPacket * 8) - MAX_BUNCH_HEADER_BITS - MAX_PACKET_TRAILER_BITS - MAX_PACKET_HEADER_BITS - MaxPacketHandlerBits;
}

void UNetConnection::InitConnection(UNetDriver* InDriver, EConnectionState InState, const FURL& InURL, int32 InConnectionSpeed, int32 InMaxPacket)
{
	Driver = InDriver;

	// We won't be sending any packets, so use a default size
	MaxPacket = (InMaxPacket == 0 || InMaxPacket > MAX_PACKET_SIZE) ? MAX_PACKET_SIZE : InMaxPacket;
	// 省略后续代码
}
enum { MAX_PACKET_SIZE = 1024 }; // MTU for the connection

这里的MaxPacket有一个硬上限MAX_PACKET_SIZE,设置为了1024，但是常见的MTU一般来说是1492，看来UE里设置的更加保守。同样由于这个MTU的限制，如果一个Bunch填充完数据发现太大了，就需要执行拆包操作:

FPacketIdRange UChannel::SendBunch( FOutBunch* Bunch, bool Merge )
{
	// 省略之前的代码
	//-----------------------------------------------------
	// Possibly split large bunch into list of smaller partial bunches
	//-----------------------------------------------------
	if( Bunch->GetNumBits() > MAX_SINGLE_BUNCH_SIZE_BITS )
	{
		uint8 *data = Bunch->GetData();
		int64 bitsLeft = Bunch->GetNumBits();
		Merge = false;

		while(bitsLeft > 0)
		{
			FOutBunch * PartialBunch = new FOutBunch(this, false);
			int64 bitsThisBunch = FMath::Min<int64>(bitsLeft, MAX_PARTIAL_BUNCH_SIZE_BITS);
			PartialBunch->SerializeBits(data, bitsThisBunch);

#if UE_NET_TRACE_ENABLED
			// Attach tracecollector of split bunch to first partial bunch
			SetTraceCollector(*PartialBunch, GetTraceCollector(*Bunch));
			SetTraceCollector(*Bunch, nullptr);
#endif

			OutgoingBunches.Add(PartialBunch);

#if !(UE_BUILD_SHIPPING || UE_BUILD_TEST)
			PartialBunch->DebugString = FString::Printf(TEXT("Partial[%d]: %s"), OutgoingBunches.Num(), *Bunch->DebugString);
#endif
		
			bitsLeft -= bitsThisBunch;
			data += (bitsThisBunch >> 3);

			UE_LOG(LogNetPartialBunch, Log, TEXT("	Making partial bunch from content bunch. bitsThisBunch: %d bitsLeft: %d"), bitsThisBunch, bitsLeft );
			
			ensure(bitsLeft == 0 || bitsThisBunch % 8 == 0); // Byte aligned or it was the last bunch
		}
	}
	else
	{
		OutgoingBunches.Add(Bunch);
	}
	// 省略后续代码
}

进行拆包的时候，会创建一个或多个PartialBunch，每个PartialBunch都会从当前Bunch里拿出最多MAX_PARTIAL_BUNCH_SIZE_BITS个bit的数据来执行填充，然后每个被填充好的PartialBunch都会以此添加到OutgoingBunches数组里。

完成Bunch的合并与拆分之后，会遍历OutgoingBunches来设置好每个可能的PartialBunch的相关属性字段，主要是处理bPartial/bPartialInitial/bPartialFinal这三个字段，bPartial代表当前包是拆分包，bPartialInitial代表当前包是拆分包里的第一个，bPartialFinal代表当前包是拆分包里的最后一个。

UE_CLOG((OutgoingBunches.Num() > 1), LogNetPartialBunch, Log, TEXT("Sending %d Bunches. Channel: %d %s"), OutgoingBunches.Num(), Bunch->ChIndex, *Describe());
for( int32 PartialNum = 0; PartialNum < OutgoingBunches.Num(); ++PartialNum)
{
	FOutBunch * NextBunch = OutgoingBunches[PartialNum];

	NextBunch->bReliable = Bunch->bReliable;
	NextBunch->bOpen = Bunch->bOpen;
	NextBunch->bClose = Bunch->bClose;
	NextBunch->CloseReason = Bunch->CloseReason;
	NextBunch->bIsReplicationPaused = Bunch->bIsReplicationPaused;
	NextBunch->ChIndex = Bunch->ChIndex;
	NextBunch->ChName = Bunch->ChName;

	if ( !NextBunch->bHasPackageMapExports )
	{
		NextBunch->bHasMustBeMappedGUIDs |= Bunch->bHasMustBeMappedGUIDs;
	}

	if (OutgoingBunches.Num() > 1)
	{
		NextBunch->bPartial = 1;
		NextBunch->bPartialInitial = (PartialNum == 0 ? 1: 0);
		NextBunch->bPartialFinal = (PartialNum == OutgoingBunches.Num() - 1 ? 1: 0);
		NextBunch->bOpen &= (PartialNum == 0);											// Only the first bunch should have the bOpen bit set
		NextBunch->bClose = (Bunch->bClose && (OutgoingBunches.Num()-1 == PartialNum)); // Only last bunch should have bClose bit set
	}

	FOutBunch *ThisOutBunch = PrepBunch(NextBunch, OutBunch, Merge); // This handles queuing reliable bunches into the ack list

	// Update Packet Range
	int32 PacketId = SendRawBunch(ThisOutBunch, Merge, GetTraceCollector(*NextBunch));
	if (PartialNum == 0)
	{
		PacketIdRange = FPacketIdRange(PacketId);
	}
	else
	{
		PacketIdRange.Last = PacketId;
	}

	// Update channel sequence count.
	Connection->LastOut = *ThisOutBunch;
	Connection->LastEnd	= FBitWriterMark( Connection->SendBuffer );
}

如果当前包是一个需要可靠传输的包，那么上面的PrepBunch会为这个Bunch分配一个在Channel内递增的序列号ChSequence，同时将这个Bunch记录在当前可靠Bunch列表OutRec的末尾：

// OUtbunch is a bunch that was new'd by the network system or NULL. It should never be one created on the stack
FOutBunch* UChannel::PrepBunch(FOutBunch* Bunch, FOutBunch* OutBunch, bool Merge)
{
	if ( Connection->ResendAllDataState != EResendAllDataState::None )
	{
		return Bunch;
	}

	// Find outgoing bunch index.
	if( Bunch->bReliable )
	{
		// Find spot, which was guaranteed available by FOutBunch constructor.
		if( OutBunch==NULL )
		{
			Bunch->Next	= NULL;
			Bunch->ChSequence = ++Connection->OutReliable[ChIndex];
			NumOutRec++;
			OutBunch = new FOutBunch(*Bunch);
			FOutBunch** OutLink = &OutRec;
			while(*OutLink) // This was rewritten from a single-line for loop due to compiler complaining about empty body for loops (-Wempty-body)
			{
				OutLink=&(*OutLink)->Next;
			}
			*OutLink = OutBunch;
		}
		else
		{
			Bunch->Next = OutBunch->Next;
			*OutBunch = *Bunch;
		}
		Connection->LastOutBunch = OutBunch;
	}
	else
	{
		OutBunch = Bunch;
		Connection->LastOutBunch = NULL;//warning: Complex code, don't mess with this!
	}

	return OutBunch;
}

执行完PrepBunch，紧接着就使用SendRawBunch将当前Bunch转化为Packet加入到发送队列里,并生成一个PacketId进行返回:

int32 UChannel::SendRawBunch(FOutBunch* OutBunch, bool Merge, const FNetTraceCollector* Collector)
{
	// Sending for checkpoints may need to send an open bunch if the actor went dormant, so allow the OpenPacketId to be set

	// Send the raw bunch.
	OutBunch->ReceivedAck = 0;
	int32 PacketId = Connection->SendRawBunch(*OutBunch, Merge, Collector);
	if( OpenPacketId.First==INDEX_NONE && OpenedLocally )
	{
		OpenPacketId = FPacketIdRange(PacketId);
	}

	if( OutBunch->bClose )
	{
		SetClosingFlag();
	}

	return PacketId;
}

这里会调用UNetConnection::SendRawBunch，这个函数负责将Bunch数据转换为真正在网络上执行发送的Packet数据，也就是二进制流:

int32 UNetConnection::SendRawBunch(FOutBunch& Bunch, bool InAllowMerge, const FNetTraceCollector* BunchCollector)
{
	ValidateSendBuffer();
	check(!Bunch.ReceivedAck);
	check(!Bunch.IsError());
	Driver->OutBunches++;
	Driver->OutTotalBunches++;
	TimeSensitive = 1;

	// Build header.
	SendBunchHeader.Reset();

	const bool bIsOpenOrClose = Bunch.bOpen || Bunch.bClose;
	const bool bIsOpenOrReliable = Bunch.bOpen || Bunch.bReliable;

	SendBunchHeader.WriteBit(bIsOpenOrClose);
	if (bIsOpenOrClose)
	{
		SendBunchHeader.WriteBit(Bunch.bOpen);
		SendBunchHeader.WriteBit(Bunch.bClose);
		if (Bunch.bClose)
		{
			uint32 Value = (uint32)Bunch.CloseReason;
			SendBunchHeader.SerializeInt(Value, (uint32)EChannelCloseReason::MAX);
		}
	}
	SendBunchHeader.WriteBit(Bunch.bIsReplicationPaused);
	SendBunchHeader.WriteBit(Bunch.bReliable);

	uint32 ChIndex = Bunch.ChIndex;
	SendBunchHeader.SerializeIntPacked(ChIndex); 

	SendBunchHeader.WriteBit(Bunch.bHasPackageMapExports);
	SendBunchHeader.WriteBit(Bunch.bHasMustBeMappedGUIDs);
	SendBunchHeader.WriteBit(Bunch.bPartial);

	if (Bunch.bReliable && !IsInternalAck())
	{
		SendBunchHeader.WriteIntWrapped(Bunch.ChSequence, MAX_CHSEQUENCE);
	}

	if (Bunch.bPartial)
	{
		SendBunchHeader.WriteBit(Bunch.bPartialInitial);
		SendBunchHeader.WriteBit(Bunch.bPartialFinal);
	}

	if (bIsOpenOrReliable)
	{
		UPackageMap::StaticSerializeName(SendBunchHeader, Bunch.ChName);
	}
	
	SendBunchHeader.WriteIntWrapped(Bunch.GetNumBits(), UNetConnection::MaxPacket * 8);

	check(!SendBunchHeader.IsError());

	// Remember start position.
	AllowMerge = InAllowMerge;
	Bunch.Time = Driver->GetElapsedTime();

	NETWORK_PROFILER(GNetworkProfiler.PushSendBunch(this, &Bunch, SendBunchHeader.GetNumBits(), Bunch.GetNumBits()));

	const int32 BunchHeaderBits = SendBunchHeader.GetNumBits();
	const int32 BunchBits = Bunch.GetNumBits();

	// If the bunch does not fit in the current packet, 
	// flush packet now so that we can report collected stats in the correct scope
	PrepareWriteBitsToSendBuffer(BunchHeaderBits, BunchBits);

	// Report bunch
	UE_NET_TRACE_END_BUNCH(OutTraceCollector, Bunch, Bunch.ChName, 0, BunchHeaderBits, BunchBits, BunchCollector);

	// Write the bits to the buffer and remember the packet id used
	Bunch.PacketId = WriteBitsToSendBufferInternal(SendBunchHeader.GetData(), BunchHeaderBits, Bunch.GetData(), BunchBits, EWriteBitsDataType::Bunch);

	// Track channels that wrote data to this packet.
	FChannelRecordImpl::PushChannelRecord(ChannelRecord, Bunch.PacketId, Bunch.ChIndex);

	// 忽略一些PackageMapClient相关的逻辑

	if (bAutoFlush)
	{
		FlushNet();
	}

	return Bunch.PacketId;
}

这里会使用SendBunchHeader这个结构来填充当前Bunch的一些元数据信息，开头的PrepareWriteBitsToSendBuffer负责先通过WritePacketHeader和WriteDummyPacketInfo在SendBuffer里写入一些无效的数据来执行占位，这些数据会在后续用最终的值来重新覆盖:

void UNetConnection::PrepareWriteBitsToSendBuffer(const int32 SizeInBits, const int32 ExtraSizeInBits)
{
	ValidateSendBuffer();

#if !UE_BUILD_SHIPPING
	// Now that the stateless handshake is responsible for initializing the packet sequence numbers,
	//	we can't allow any packets to be written to the send buffer until after this has completed
	if (CVarRandomizeSequence.GetValueOnAnyThread() > 0)
	{
		checkf(!Handler.IsValid() || Handler->IsFullyInitialized(), TEXT("Attempted to write to send buffer before packet handler was fully initialized. Connection: %s"), *Describe());
	}
#endif

	const int32 TotalSizeInBits = SizeInBits + ExtraSizeInBits;

	// Flush if we can't add to current buffer
	if ( TotalSizeInBits > GetFreeSendBufferBits() )
	{
		FlushNet();
	}

	// If this is the start of the queue, make sure to add the packet id
	if ( SendBuffer.GetNumBits() == 0 && !IsInternalAck() )
	{
		// Write Packet Header, before sending the packet we will go back and rewrite the data
		WritePacketHeader(SendBuffer);

		// Pre-write the bits for the packet info
		WriteDummyPacketInfo(SendBuffer);

		// We do not allow the first bunch to merge with the ack data as this will "revert" the ack data.
		AllowMerge = false;
	
		// Update stats for PacketIdBits and ackdata (also including the data used for packet RTT and saturation calculations)
		int64 BitsWritten = SendBuffer.GetNumBits();
		NumPacketIdBits += FNetPacketNotify::SequenceNumberT::SeqNumberBits;
		NumAckBits += BitsWritten - FNetPacketNotify::SequenceNumberT::SeqNumberBits;

		// Report stats to profiler
		NETWORK_PROFILER( GNetworkProfiler.TrackSendAck( NumAckBits, this ) );

		ValidateSendBuffer();
	}
}

占据好头部空间之后，再使用WriteBitsToSendBufferInternal将SendBunchHeader的内容与Bunch的内容组合起来，放到当前UNetConnection的SendBuffer的后面:

int32 UNetConnection::WriteBitsToSendBufferInternal( 
	const uint8 *	Bits, 
	const int32		SizeInBits, 
	const uint8 *	ExtraBits, 
	const int32		ExtraSizeInBits,
	EWriteBitsDataType DataType)
{
	// Remember start position in case we want to undo this write, no meaning to undo the header write as this is only used to pop bunches and the header should not count towards the bunch
	// Store this after the possible flush above so we have the correct start position in the case that we do flush
	LastStart = FBitWriterMark( SendBuffer );

	// Add the bits to the queue
	if ( SizeInBits )
	{
		SendBuffer.SerializeBits( const_cast< uint8* >( Bits ), SizeInBits );
		ValidateSendBuffer();
	}

	// Add any extra bits
	if ( ExtraSizeInBits )
	{
		SendBuffer.SerializeBits( const_cast< uint8* >( ExtraBits ), ExtraSizeInBits );
		ValidateSendBuffer();
	}

	const int32 RememberedPacketId = OutPacketId;

	switch ( DataType )
	{
		case EWriteBitsDataType::Bunch:
			NumBunchBits += SizeInBits + ExtraSizeInBits;
			break;
		default:
			break;
	}

	// Flush now if we are full
	if (GetFreeSendBufferBits() == 0
#if !UE_BUILD_SHIPPING
		|| CVarForceNetFlush.GetValueOnAnyThread() != 0
#endif
		)
	{
		FlushNet();
	}

	return RememberedPacketId;
}

由于MTU的限制，SendBuffer不能无限制的加入数据，所以这个SendBuffer也会以当前MaxPacket=1024的大小来初始化内部的Buffer:

void UNetConnection::InitSendBuffer()
{
	check(MaxPacket > 0);

	int32 FinalBufferSize = (MaxPacket * 8) - MaxPacketHandlerBits;

	// Initialize the one outgoing buffer.
	if (FinalBufferSize == SendBuffer.GetMaxBits())
	{
		// Reset all of our values to their initial state without a malloc/free
		SendBuffer.Reset();
	}
	else
	{
		// First time initialization needs to allocate the buffer
		SendBuffer = FBitWriter(FinalBufferSize);
	}

	HeaderMarkForPacketInfo.Reset();

	ResetPacketBitCounts();

	ValidateSendBuffer();
}

如果发现加入了一个Bunch之后剩余空间已经无法再放入新的Bunch，那么这里会强制调用FlushNet，将当前的SendBuffer里的数据发送出去。FlushNet的内部逻辑可以大概精简为下面的代码:

void UNetConnection::FlushNet(bool bIgnoreSimulation)
{
	check(Driver);

	// Update info.
	ValidateSendBuffer();
	LastEnd = FBitWriterMark();
	TimeSensitive = 0;

	const double PacketSentTimeInS = FPlatformTime::Seconds();

	// Write the UNetConnection-level termination bit
	SendBuffer.WriteBit(1);

	// Refresh outgoing header with latest data
	if ( !IsInternalAck() )
	{
		// if we update ack, we also update received ack associated with outgoing seq
		// so we know how many ack bits we need to write (which is updated in received packet)
		WritePacketHeader(SendBuffer);

		WriteFinalPacketInfo(SendBuffer, PacketSentTimeInS);
	}

	// Send now.
	// Checked in FlushNet() so each child class doesn't have to implement this
	if (Driver->IsNetResourceValid())
	{
		LowLevelSend(SendBuffer.GetData(), SendBuffer.GetNumBits(), Traits);
	}
		

	// Update stuff.
	const int32 Index = OutPacketId & (UE_ARRAY_COUNT(OutLagPacketId)-1);

	// Remember the actual time this packet was sent out, so we can compute ping when the ack comes back
	OutLagPacketId[Index]			= OutPacketId;
	OutLagTime[Index]				= PacketSentTimeInS;
	OutBytesPerSecondHistory[Index]	= FMath::Min(OutBytesPerSecond / 1024, 255);
	

	// Increase outgoing sequence number
	if (!IsInternalAck())
	{
		PacketNotify.CommitAndIncrementOutSeq();
	}

	// Make sure that we always push an ChannelRecordEntry for each transmitted packet even if it is empty
	FChannelRecordImpl::PushPacketId(ChannelRecord, OutPacketId);

	

	++OutPackets;
	++OutTotalPackets;
	Driver->OutPackets++;
	Driver->OutTotalPackets++;

	//Record the first packet time in the histogram
	if (!bFlushedNetThisFrame)
	{
		double LastPacketTimeDiffInMs = (Driver->GetElapsedTime() - LastSendTime) * 1000.0;
		NetConnectionHistogram.AddMeasurement(LastPacketTimeDiffInMs);
	}

	LastSendTime = Driver->GetElapsedTime();
	++OutPacketId; 
}

这里的SendBuffer.WriteBit(1)负责在SendBuffer的末尾添加一个值为1的bit，作为当前Packet的终止符。由于UDP包都是按照字节传输的，所以在SendBuffer.WriteBit(1)之后，还会在SendBuffer的末尾添加0个或者多个bit，直到SendBuffer的总bit数是8的倍数。

这里的WritePacketHeader负责添加可靠传输的序列号相关信息，这个函数的细节将在后续的可靠传输中介绍。而WriteFinalPacketInfo负责添加服务器与客户端之间的网络延迟信息，数据量为20bit，其实就是在发包的时候带上时间戳，并记录在本地的OutLagTime数组里。

这两个函数写入的数据都会在当前SendBuffer的头部，刚好对应之前PrepareWriteBitsToSendBuffer在SendBuffer里占据好的位置。同时这里还会在SendBuffer的末尾添加一个值为1的bit，作为当前Packet的终止符。这两个函数都执行完之后SendBuffer里就有最终的二进制，整个Packet的格式如下:

最后使用LowLevelSend将SendBuffer将数据发送出去， 这里的LowLevelSend最终会调用到平台相关的Socket::SendTo,在Windows平台就是FSocketWindows，在Linux平台就是FSocketBSD:

void UIpConnection::LowLevelSend(void* Data, int32 CountBits, FOutPacketTraits& Traits)
{
	// 省略很多分支条件
	// Send to remote.
	FSocketSendResult SendResult;
	CLOCK_CYCLES(Driver->SendCycles);

	if ( CountBytes > MaxPacket )
	{
		UE_LOG( LogNet, Warning, TEXT( "UIpConnection::LowLevelSend: CountBytes > MaxPacketSize! Count: %i, MaxPacket: %i %s" ), CountBytes, MaxPacket, *Describe() );
	}

	FPacketAudit::NotifyLowLevelSend((uint8*)DataToSend, CountBytes, CountBits);

	if (CountBytes > 0)
	{
		const bool bNotifyOnSuccess = (SocketErrorDisconnectDelay > 0.f) && (SocketError_SendDelayStartTime != 0.f);
		FSocket* CurSocket = GetSocket();

		if (CVarNetIpConnectionUseSendTasks.GetValueOnAnyThread() != 0)
		{
			DECLARE_CYCLE_STAT(TEXT("IpConnection SendTo task"), STAT_IpConnection_SendToTask, STATGROUP_TaskGraphTasks);

			FGraphEventArray Prerequisites;
			if (LastSendTask.IsValid())
			{
				Prerequisites.Add(LastSendTask);
			}

			ISocketSubsystem* const SocketSubsystem = Driver->GetSocketSubsystem();
			
			LastSendTask = FFunctionGraphTask::CreateAndDispatchWhenReady([this, Packet = TArray<uint8>(DataToSend, CountBytes), SocketSubsystem, bNotifyOnSuccess]
			{
				FSocket* CurSocket = GetSocket();

				if (CurSocket != nullptr)
				{
					bool bWasSendSuccessful = false;
					UIpConnection::FSocketSendResult Result;

					{
						SCOPE_CYCLE_COUNTER(STAT_IpConnection_SendToSocket);
						bWasSendSuccessful = CurSocket->SendTo(Packet.GetData(), Packet.Num(), Result.BytesSent, *RemoteAddr);
					}
				}
			},
			GET_STATID(STAT_IpConnection_SendToTask), &Prerequisites);
		}
	}
}

这里的SendTo依然是对平台的网络接口的封装，只有具体的平台子类里才能调用到最终的操作系统API:

bool FSocketBSD::SendTo(const uint8* Data, int32 Count, int32& BytesSent, const FInternetAddr& Destination)
{
	// TODO: Consider converting IPv4 addresses to v6 when needed
	if (Destination.GetProtocolType() != GetProtocol())
	{
		UE_LOG(LogSockets, Warning, TEXT("Destination protocol of '%s' does not match protocol: '%s' for address: '%s'"),
			*Destination.GetProtocolType().ToString(), *GetProtocol().ToString(), *Destination.ToString(true));
		return false;
	}

	const FInternetAddrBSD& BSDAddr = static_cast<const FInternetAddrBSD&>(Destination);
	// Write the data and see how much was written
	BytesSent = sendto(Socket, (const char*)Data, Count, SendFlags, (const sockaddr*)&(BSDAddr.Addr), BSDAddr.GetStorageSize());

//	NETWORK_PROFILER(FSocket::SendTo(Data,Count,BytesSent,Destination));

	bool Result = BytesSent >= 0;
	if (Result)
	{
		LastActivityTime = FPlatformTime::Seconds();
	}
	return Result;
}
/**
 * Implements a Windows/BSD network socket.
 */
class FSocketWindows
	: public FSocketBSD
{
public:
	FSocketWindows(SOCKET InSocket, ESocketType InSocketType, const FString& InSocketDescription, const FName& InSocketProtocol, ISocketSubsystem* InSubsystem)
		: FSocketBSD(InSocket, InSocketType, InSocketDescription, InSocketProtocol, InSubsystem)
	{ }

	// FSocketBSD overrides

	virtual bool Shutdown(ESocketShutdownMode Mode) override;
	virtual bool SetIpPktInfo(bool bEnable) override;
	virtual bool RecvFromWithPktInfo(uint8* Data, int32 BufferSize, int32& BytesRead, FInternetAddr& Source, FInternetAddr& Destination, ESocketReceiveFlags::Type Flags = ESocketReceiveFlags::None) override;

protected:
	LPFN_WSARECVMSG WSARecvMsg = nullptr;
};

注意到这里为了尽可能的维持统一的网路IO模型，网络底层使用的都是最基础的IO接口，而没有去使用IO多路复用以及IO完成端口等高级特性。因为目前UE的设计里单服务器的客户端连接一般不会超过200，所以最简单的同步通信模型也能满足需求。

消息接收

消息接收由Tick驱动,当World::Tick的时候，会出发对应的NetDriver::TickDispatch:

void UNetDriver::RegisterTickEvents(class UWorld* InWorld)
{
	if (InWorld)
	{
		TickDispatchDelegateHandle  = InWorld->OnTickDispatch ().AddUObject(this, &UNetDriver::TickDispatch);
		PostTickDispatchDelegateHandle	= InWorld->OnPostTickDispatch().AddUObject(this, &UNetDriver::PostTickDispatch);
		TickFlushDelegateHandle     = InWorld->OnTickFlush    ().AddUObject(this, &UNetDriver::TickFlush);
		PostTickFlushDelegateHandle		= InWorld->OnPostTickFlush	 ().AddUObject(this, &UNetDriver::PostTickFlush);
	}
}

在这个TickDispatch里，会使用一个迭代器来访问当前已经接收到的数据:

void UIpNetDriver::TickDispatch(float DeltaTime)
{
	LLM_SCOPE(ELLMTag::Networking);

	Super::TickDispatch( DeltaTime );

#if !UE_BUILD_SHIPPING
	PauseReceiveEnd = (PauseReceiveEnd != 0.f && PauseReceiveEnd - (float)FPlatformTime::Seconds() > 0.f) ? PauseReceiveEnd : 0.f;

	if (PauseReceiveEnd != 0.f)
	{
		return;
	}
#endif

	// Set the context on the world for this driver's level collection.
	const int32 FoundCollectionIndex = World ? World->GetLevelCollections().IndexOfByPredicate([this](const FLevelCollection& Collection)
	{
		return Collection.GetNetDriver() == this;
	}) : INDEX_NONE;

	FScopedLevelCollectionContextSwitch LCSwitch(FoundCollectionIndex, World);


	DDoS.PreFrameReceive(DeltaTime);

	ISocketSubsystem* SocketSubsystem = GetSocketSubsystem();
	bool bRetrieveTimestamps = CVarNetUseRecvTimestamps.GetValueOnAnyThread() != 0;

	// Process all incoming packets
	for (FPacketIterator It(this); It; ++It)
	{
		FReceivedPacketView ReceivedPacket;
		FInPacketTraits& ReceivedTraits = ReceivedPacket.Traits;
		bool bOk = It.GetCurrentPacket(ReceivedPacket);
		const TSharedRef<const FInternetAddr> FromAddr = ReceivedPacket.Address.ToSharedRef();
		UNetConnection* Connection = nullptr;
		UIpConnection* const MyServerConnection = GetServerConnection();
		// 暂时省略Packet的处理逻辑
	}
}

在这个FPacketIterator的构造函数和迭代器函数里会使用AdvanceCurrentPacket来获取下一个Packet:

FPacketIterator(UIpNetDriver* InDriver, FRecvMulti* InRMState, double InStartReceiveTime, bool bInCheckReceiveTime)
{
	if (!bUseRecvMulti && SocketSubsystem != nullptr)
	{
		CurrentPacket.Address = SocketSubsystem->CreateInternetAddr();
	}

	AdvanceCurrentPacket();
}

FORCEINLINE FPacketIterator& operator++()
{
	IterationCount++;
	AdvanceCurrentPacket();

	return *this;
}
/**
	* Advances the current packet to the next iteration
	*/
void AdvanceCurrentPacket()
{
	// 省略很多代码
	if (bUseRecvMulti)
	{
		// 忽略多线程接收数据

		// At this point, bBreak will be set, or RecvMultiPacketCount will be > 0
	}
	else
	{
		bBreak = !ReceiveSinglePacket();
	}
}

AdvanceCurrentPacket内部会根据是否开启了多线程接收数据来执行不同的逻辑，其多线程开关为bUseRecvMulti，默认情况下这个开关是关的，我们只关心在主线程里接收数据的情况:

/**
	* Receives a single packet from the network socket, outputting to the CurrentPacket buffer.
	*
	* @return				Whether or not a packet or an error was successfully received
	*/
bool ReceiveSinglePacket()
{
	bool bReceivedPacketOrError = false;

	CurrentPacket.bRecvSuccess = false;
	CurrentPacket.Data.SetNumUninitialized(0, false);

	if (CurrentPacket.Address.IsValid())
	{
		CurrentPacket.Address->SetAnyAddress();
	}

	CurrentPacket.PacketTimestamp = 0.0;
	CurrentPacket.Error = SE_NO_ERROR;

	while (true)
	{
		bReceivedPacketOrError = false;

		if (SocketReceiveThreadRunnable != nullptr)
		{
			// 省略多线程接收数据的部分
		}
		else if (Driver->GetSocket() != nullptr && SocketSubsystem != nullptr)
		{
			SCOPE_CYCLE_COUNTER(STAT_IpNetDriver_RecvFromSocket);

			int32 BytesRead = 0;
			bool bReceivedPacket = Driver->GetSocket()->RecvFrom(CurrentPacket.Data.GetData(), MAX_PACKET_SIZE, BytesRead, *CurrentPacket.Address);

			CurrentPacket.bRecvSuccess = bReceivedPacket;
			bReceivedPacketOrError = bReceivedPacket;

			if (bReceivedPacket)
			{
				// Fixed allocator, so no risk of realloc from copy-then-resize
				CurrentPacket.Data.SetNumUninitialized(BytesRead, false);
			}
			else
			{
				// 忽略错误处理代码
			}

		}

		// While loop only exists to allow 'continue' for DDoS and invalid packet code, above
		break;
	}

	return bReceivedPacketOrError;
}

在ReceiveSinglePacket内部会通过Driver->GetSocket()->RecvFrom去从UDP端口里接收一个Packet到CurrentPacket.Data中。外部再使用GetCurrentPacket来尝试获取接收到的Packet的DataView:

/**
	* Retrieves the packet information from the current iteration. Avoid calling more than once, per iteration.
	*
	* @param OutPacket		Outputs a view to the received packet data
	* @return				Returns whether or not receiving was successful for the current packet
	*/
bool GetCurrentPacket(FReceivedPacketView& OutPacket)
{
	bool bRecvSuccess = false;

	if (bUseRecvMulti)
	{
		RMState->GetPacket(RecvMultiIdx, OutPacket);
		bRecvSuccess = true;
	}
	else
	{
		OutPacket.DataView = {CurrentPacket.Data.GetData(), CurrentPacket.Data.Num(), ECountUnits::Bytes};
		OutPacket.Error = CurrentPacket.Error;
		OutPacket.Address = CurrentPacket.Address;
		bRecvSuccess = CurrentPacket.bRecvSuccess;
	}

	return bRecvSuccess;
}

当接收到一个有效的FReceivedPacketView之后，会通知到对应的Connection里去执行OnReceiveRawPacket函数。这里寻找对应的Connection的过程很简单，如果当前是客户端则直接使用对应的服务端连接MyServerConnection，如果是服务端则通过消息的来源地址去查找MappedClientConnections:

// Figure out which socket the received data came from.
if (MyServerConnection)
{
	if (MyServerConnection->RemoteAddr->CompareEndpoints(*FromAddr))
	{
		Connection = MyServerConnection;
	}
	else
	{
		UE_LOG(LogNet, Warning, TEXT("Incoming ip address doesn't match expected server address: Actual: %s Expected: %s"),
			*FromAddr->ToString(true),
			MyServerConnection->RemoteAddr.IsValid() ? *MyServerConnection->RemoteAddr->ToString(true) : TEXT("Invalid"));
	}
}

if (Connection == nullptr)
{
	UNetConnection** Result = MappedClientConnections.Find(FromAddr);

	if (Result != nullptr)
	{
		UNetConnection* ConnVal = *Result;

		if (ConnVal != nullptr)
		{
			Connection = ConnVal;
		}
		else
		{
			ReceivedTraits.bFromRecentlyDisconnected = true;
		}
	}
	check(Connection == nullptr || CastChecked<UIpConnection>(Connection)->RemoteAddr->CompareEndpoints(*FromAddr));
}
bool bIgnorePacket = false;

// If we didn't find a client connection, maybe create a new one.
if (Connection == nullptr)
{
	// 忽略创建新连接的代码
}

// Send the packet to the connection for processing.
if (Connection != nullptr && !bIgnorePacket)
{
	if (bRetrieveTimestamps)
	{
		It.GetCurrentPacketTimestamp(Connection);
	}

	Connection->ReceivedRawPacket((uint8*)ReceivedPacket.DataView.GetData(), ReceivedPacket.DataView.NumBytes());
}

在UNetConnection::ReceivedRawPacket首先会使用Handler来对接收到的数据做一遍处理，这里可能会有一些解密解压缩相关的流程：

void UNetConnection::ReceivedRawPacket( void* InData, int32 Count )
{
#if !UE_BUILD_SHIPPING
	// Add an opportunity for the hook to block further processing
	bool bBlockReceive = false;

	ReceivedRawPacketDel.ExecuteIfBound(InData, Count, bBlockReceive);

	if (bBlockReceive)
	{
		return;
	}
#endif

#if DO_ENABLE_NET_TEST
	// Opportunity for packet loss burst simulation to drop the incoming packet.
	if (Driver && Driver->IsSimulatingPacketLossBurst())
	{
		return;
	}
#endif

	uint8* Data = (uint8*)InData;

	if (Handler.IsValid())
	{
		const ProcessedPacket UnProcessedPacket = Handler->Incoming(Data, Count);

		if (!UnProcessedPacket.bError)
		{
			Count = FMath::DivideAndRoundUp(UnProcessedPacket.CountBits, 8);

			if (Count > 0)
			{
				Data = UnProcessedPacket.Data;
			}
			// This packed has been consumed
			else
			{
				return;
			}
		}
	}


	// Handle an incoming raw packet from the driver.
	UE_LOG(LogNetTraffic, Verbose, TEXT("%6.3f: Received %i"), FPlatformTime::Seconds() - GStartTime, Count );
	int32 PacketBytes = Count + PacketOverhead;
	InBytes += PacketBytes;
	InTotalBytes += PacketBytes;
	++InPackets;
	++InTotalPackets;

	if (Driver)
	{
		Driver->InBytes += PacketBytes;
		Driver->InTotalBytes += PacketBytes;
		Driver->InPackets++;
		Driver->InTotalPackets++;
	}

	if (Count > 0)
	{
		uint8 LastByte = Data[Count-1];

		if (LastByte != 0)
		{
			int32 BitSize = (Count * 8) - 1;

			// Bit streaming, starts at the Least Significant Bit, and ends at the MSB.
			while (!(LastByte & 0x80))
			{
				LastByte *= 2;
				BitSize--;
			}


			FBitReader Reader(Data, BitSize);

			// Set the network version on the reader
			Reader.SetEngineNetVer( EngineNetworkProtocolVersion );
			Reader.SetGameNetVer( GameNetworkProtocolVersion );

			if (Handler.IsValid())
			{
				Handler->IncomingHigh(Reader);
			}

			if (Reader.GetBitsLeft() > 0)
			{
				ReceivedPacket(Reader);

				// Check if the out of order packet cache needs flushing
				FlushPacketOrderCache();
			}
		}
	}
}

在Handle处理完之后，还有一道非常重要的工序，即寻找当前Packet的最后一个Bit。在前面的消息发送部分我们提到过UE会在Packet的末尾添加一个值为1的bit，但是UDP发送数据的时候数据的粒度是字节，所以如果SendBuffer无法组成完整字节的情况下，会在这个bit之后用0去填充数据。所以在接收到以字节为单位的数据的时候，需要重新寻找到最后的1的bit作为边界，也就是上面操作LastByte的While循环部分的逻辑，用来计算真正的数据bit大小BitSize。

获取了真正有效的数据之后，会以这个数据构造出FBitReader，然后调用ReceivePacket去处理。在ReceivePacket的开头会首先通过PacketNotify.ReadHeader将packet的头部数据读取出来，也就是FNotificationHeader部分：

// Read packet header
FNetPacketNotify::FNotificationHeader Header;
if (!PacketNotify.ReadHeader(Header, Reader))
{
	CLOSE_CONNECTION_DUE_TO_SECURITY_VIOLATION(this, ESecurityEvent::Malformed_Packet, TEXT("Failed to read PacketHeader"));
	return;
}

// 暂时省略一些处理乱序接收的代码

// Extra information associated with the header (read only after acks have been processed)
if (PacketSequenceDelta > 0 && !ReadPacketInfo(Reader, bHasPacketInfoPayload))
{
	CLOSE_CONNECTION_DUE_TO_SECURITY_VIOLATION(this, ESecurityEvent::Malformed_Packet, TEXT("Failed to read PacketHeader"));
	return;
}

读取完这个FNotificationHeader头部数据之后，会利用这个头部里携带的序列号信息来判断当前Packet是否乱序，如果是乱序包可能会不处理直接return。这里我们先不去关注有序接收Packet的细节，先假设我们当前的Packet就是下一个需要的Packet。

确认是有序Packet被接收之后， 后面的ReadPacketInfo负责读取之前发包的时候通过WriteFinalPacketInfo写入的一些时间戳信息, 根据本地时间与OutLagTime里记录的发包时间来计算网络延迟RTT，从而去更新Ping值，

bool UNetConnection::ReadPacketInfo(FBitReader& Reader, bool bHasPacketInfoPayload)
{
	// 省略解析PacketInfo的代码
	// Update ping
	// At this time we have updated OutAckPacketId to the latest received ack.
	const int32 Index = OutAckPacketId & (UE_ARRAY_COUNT(OutLagPacketId)-1);

	if ( OutLagPacketId[Index] == OutAckPacketId )
	{
		OutLagPacketId[Index] = -1;		// Only use the ack once

		double PacketReceiveTime = 0.0;
		FTimespan& RecvTimespan = LastOSReceiveTime.Timestamp;

		if (!RecvTimespan.IsZero() && Driver != nullptr && CVarPingUsePacketRecvTime.GetValueOnAnyThread())
		{
			if (bIsOSReceiveTimeLocal)
			{
				PacketReceiveTime = RecvTimespan.GetTotalSeconds();
			}
			else if (ISocketSubsystem* SocketSubsystem = Driver->GetSocketSubsystem())
			{
				PacketReceiveTime = SocketSubsystem->TranslatePacketTimestamp(LastOSReceiveTime);
			}
		}


		// use FApp's time because it is set closer to the beginning of the frame - we don't care about the time so far of the current frame to process the packet
		const double CurrentTime = (PacketReceiveTime != 0.0 ? PacketReceiveTime : FApp::GetCurrentTime());
		const double RTT		 = (CurrentTime - OutLagTime[Index] ) - ( CVarPingExcludeFrameTime.GetValueOnAnyThread() ? ServerFrameTime : 0.0 );
		const double NewLag		 = FMath::Max( RTT, 0.0 );

		//UE_LOG( LogNet, Warning, TEXT( "Out: %i, InRemote: %i, Saturation: %f" ), OutBytesPerSecondHistory[Index], RemoteInKBytesPerSecond, RemoteSaturation );

		LagAcc += NewLag;
		LagCount++;

		if (PlayerController)
		{
			PlayerController->UpdatePing(NewLag);
		}

		if (NetworkCongestionControl.IsSet())
		{
			NetworkCongestionControl.GetValue().OnAck({ CurrentTime, OutAckPacketId });
		}
	}
}

上述内容就是解析并处理Packet头部信息的相关逻辑，当判断了当前Packet是非乱序Packet之后，接下来就是处理其内部的Bunch数据，由于一个Packet内可能有零个或者一到多个Bunch存在，因此这里使用的是While循环,每次解析出来一个Bunch之后都会通知对应的Channel来执行ReceivedRawBunch:

// Disassemble and dispatch all bunches in the packet.
while( !Reader.AtEnd() && State!=USOCK_Closed )
{
	// For demo backwards compatibility, old replays still have this bit
	if (IsInternalAck() && EngineNetworkProtocolVersion < EEngineNetworkVersionHistory::HISTORY_ACKS_INCLUDED_IN_HEADER)
	{
		const bool IsAckDummy = Reader.ReadBit() == 1u;
	}

	// Parse the bunch.
	int32 StartPos = Reader.GetPosBits();
	
	// Process Received data
	{
		// Parse the incoming data.
		FInBunch Bunch( this );
		int32 IncomingStartPos		= Reader.GetPosBits();
		uint8 bControl				= Reader.ReadBit();
		Bunch.PacketId				= InPacketId;
		Bunch.bOpen					= bControl ? Reader.ReadBit() : 0;
		Bunch.bClose				= bControl ? Reader.ReadBit() : 0;
		
		if (Bunch.EngineNetVer() < HISTORY_CHANNEL_CLOSE_REASON)
		{
			const uint8 bDormant = Bunch.bClose ? Reader.ReadBit() : 0;
			Bunch.CloseReason = bDormant ? EChannelCloseReason::Dormancy : EChannelCloseReason::Destroyed;
		}
		else
		{
			Bunch.CloseReason = Bunch.bClose ? (EChannelCloseReason)Reader.ReadInt((uint32)EChannelCloseReason::MAX) : EChannelCloseReason::Destroyed;
		}

		Bunch.bIsReplicationPaused  = Reader.ReadBit();
		Bunch.bReliable				= Reader.ReadBit();

		if (Bunch.EngineNetVer() < HISTORY_MAX_ACTOR_CHANNELS_CUSTOMIZATION)
		{
			static const int OLD_MAX_ACTOR_CHANNELS = 10240;
			Bunch.ChIndex = Reader.ReadInt(OLD_MAX_ACTOR_CHANNELS);
		}
		else
		{
			uint32 ChIndex;
			Reader.SerializeIntPacked(ChIndex);

			if (ChIndex >= (uint32)MaxChannelSize)
			{
				CLOSE_CONNECTION_DUE_TO_SECURITY_VIOLATION(this, ESecurityEvent::Malformed_Packet, TEXT("Bunch channel index exceeds channel limit"));
				return;
			}

			Bunch.ChIndex = ChIndex;
		}
		UChannel* Channel = Channels[Bunch.ChIndex];
		// 省略后续的解析bunch逻辑
		// Dispatch the raw, unsequenced bunch to the channel.
		bool bLocalSkipAck = false;
		Channel->ReceivedRawBunch( Bunch, bLocalSkipAck ); //warning: May destroy channel.
		if ( bLocalSkipAck )
		{
			bSkipAck = true;
		}
		Driver->InBunches++;
		Driver->InTotalBunches++;
	}
}

这里获取对应的Channel的时候使用的是Bunch里的ChIndex，同一个Channel在服务器客户端的ChIndex都是一样的。

在UChannel::ReceivedRawBunch会根据当前Bunch是否是可靠消息来做不同的处理，这里我们先忽略掉这个可靠消息接收的部分，重点来看这里是如何处理小包的拆包和大包的合并的:

void UChannel::ReceivedRawBunch( FInBunch & Bunch, bool & bOutSkipAck )
{
	SCOPE_CYCLE_COUNTER(Stat_ChannelReceivedRawBunch);

	SCOPED_NAMED_EVENT(UChannel_ReceivedRawBunch, FColor::Green);
	// Immediately consume the NetGUID portion of this bunch, regardless if it is partial or reliable.
	// NOTE - For replays, we do this even earlier, to try and load this as soon as possible, in case there is an issue creating the channel
	// If a replay fails to create a channel, we want to salvage as much as possible
	if ( Bunch.bHasPackageMapExports && !Connection->IsInternalAck() )
	{
		Cast<UPackageMapClient>( Connection->PackageMap )->ReceiveNetGUIDBunch( Bunch );

		if ( Bunch.IsError() )
		{
			UE_LOG( LogNetTraffic, Error, TEXT( "UChannel::ReceivedRawBunch: Bunch.IsError() after ReceiveNetGUIDBunch. ChIndex: %i" ), ChIndex );
			return;
		}
	}

	if ( Connection->IsInternalAck() && Broken )
	{
		return;
	}

	check(Connection->Channels[ChIndex]==this);

	if ( Bunch.bReliable && Bunch.ChSequence != Connection->InReliable[ChIndex] + 1 )
	{
		// 先暂时忽略掉可靠消息的乱序接收处理
	}
	else
	{
		bool bDeleted = ReceivedNextBunch( Bunch, bOutSkipAck );

		if ( Bunch.IsError() )
		{
			UE_LOG( LogNetTraffic, Error, TEXT( "UChannel::ReceivedRawBunch: Bunch.IsError() after ReceivedNextBunch 1" ) );
			return;
		}

		if (bDeleted)
		{
			return;
		}
		
		// Dispatch any waiting bunches.
		while( InRec )
		{
			// We shouldn't hit this path on 100% reliable connections
			check( !Connection->IsInternalAck() );

			if( InRec->ChSequence!=Connection->InReliable[ChIndex]+1 )
				break;
			UE_LOG(LogNetTraffic, Log, TEXT("      Channel %d Unleashing queued bunch"), ChIndex );
			FInBunch* Release = InRec;
			InRec = InRec->Next;
			NumInRec--;
			
			// Just keep a local copy of the bSkipAck flag, since these have already been acked and it doesn't make sense on this context
			// Definitely want to warn when this happens, since it's really not possible
			bool bLocalSkipAck = false;

			bDeleted = ReceivedNextBunch( *Release, bLocalSkipAck );

			if ( bLocalSkipAck )
			{
				UE_LOG( LogNetTraffic, Warning, TEXT( "UChannel::ReceivedRawBunch: bLocalSkipAck == true for already acked packet" ) );
			}

			if ( Bunch.IsError() )
			{
				UE_LOG( LogNetTraffic, Error, TEXT( "UChannel::ReceivedRawBunch: Bunch.IsError() after ReceivedNextBunch 2" ) );
				return;
			}

			delete Release;
			if (bDeleted)
			{
				return;
			}
			//AssertInSequenced();
		}
	}
}

在这个函数里会使用ReceivedNextBunch来从当前的Bunch解析出来HandleBunch，这个HandleBunch代表一个完整的逻辑包。如果当前包不是被拆分的包的话这个HandleBunch就是传入的Bunch，如果当前包是拆分包则需要处理大包的子包合并问题:

bool UChannel::ReceivedNextBunch( FInBunch & Bunch, bool & bOutSkipAck )
{
	// We received the next bunch. Basically at this point:
	//	-We know this is in order if reliable
	//	-We dont know if this is partial or not
	// If its not a partial bunch, of it completes a partial bunch, we can call ReceivedSequencedBunch to actually handle it
	
	// Note this bunch's retirement.
	if ( Bunch.bReliable )
	{
		// Reliables should be ordered properly at this point
		check( Bunch.ChSequence == Connection->InReliable[Bunch.ChIndex] + 1 );

		Connection->InReliable[Bunch.ChIndex] = Bunch.ChSequence;
	}

	FInBunch* HandleBunch = &Bunch;
	if (Bunch.bPartial)
	{
		HandleBunch = NULL;
		// 暂时省略大包的合并相关代码
	}

	// 省略后续代码
}

现在来研究一下UChannel::ReceivedNextBunch里对于PartialBunch是如何处理的，首先会判断当前Bunch是否是大包拆分之后的第一个包，也就是bPartialInitial这个标记位为true。如果是第一个拆分包的话，会检查当前是否已经在处理其他的拆分包，这个已经在处理的拆分包会记录在当前Channel的InPartialBunch字段上。如果已经有了InPartialBunch，那么需要删除这个InPartialBunch，并记录一些错误日志，因为当前的Bunch肯定已经乱序接收了:

if (Bunch.bPartialInitial)
{
	// Create new InPartialBunch if this is the initial bunch of a new sequence.

	if (InPartialBunch != NULL)
	{
		if (!InPartialBunch->bPartialFinal)
		{
			if ( InPartialBunch->bReliable )
			{
				if ( Bunch.bReliable )
				{
					UE_LOG(LogNetPartialBunch, Warning, TEXT("Reliable partial trying to destroy reliable partial 1. %s"), *Describe());
					Bunch.SetError();
					return false;
				}
				UE_LOG(LogNetPartialBunch, Log, TEXT( "Unreliable partial trying to destroy reliable partial 1") );
				bOutSkipAck = true;
				return false;
			}

			// We didn't complete the last partial bunch - this isn't fatal since they can be unreliable, but may want to log it.
			UE_LOG(LogNetPartialBunch, Verbose, TEXT("Incomplete partial bunch. Channel: %d ChSequence: %d"), InPartialBunch->ChIndex, InPartialBunch->ChSequence);
		}
		
		delete InPartialBunch;
		InPartialBunch = NULL;
	}

	InPartialBunch = new FInBunch(Bunch, false);
	if ( !Bunch.bHasPackageMapExports && Bunch.GetBitsLeft() > 0 )
	{
		if ( Bunch.GetBitsLeft() % 8 != 0 )
		{
			UE_LOG(LogNetPartialBunch, Warning, TEXT("Corrupt partial bunch. Initial partial bunches are expected to be byte-aligned. BitsLeft = %u. %s"), Bunch.GetBitsLeft(), *Describe());
			Bunch.SetError();
			return false;
		}

		InPartialBunch->AppendDataFromChecked( Bunch.GetDataPosChecked(), Bunch.GetBitsLeft() );

		LogPartialBunch(TEXT("Received new partial bunch."), Bunch, *InPartialBunch);
	}
	else
	{
		LogPartialBunch(TEXT("Received New partial bunch. It only contained NetGUIDs."), Bunch, *InPartialBunch);
	}
}

完成了异常情况的处理之后，会以当前的Bunch来创建一个新的InPartialBunch,同时将当前Bunch里的数据添加到InPartialBunch的后面。

如果当前的Bunch并不是InitialBunch,那么就会将这个Bunch的数据拼接到InPartialBunch的后面，但是这个Bunch拼接是有条件的，要求两个Bunch的bReliable是一致的，且Bunch里记录的序列号要求一致。:

// Merge in next partial bunch to InPartialBunch if:
//	-We have a valid InPartialBunch
//	-The current InPartialBunch wasn't already complete
//  -ChSequence is next in partial sequence
//	-Reliability flag matches

bool bSequenceMatches = false;
if (InPartialBunch)
{
	const bool bReliableSequencesMatches = Bunch.ChSequence == InPartialBunch->ChSequence + 1;
	const bool bUnreliableSequenceMatches = bReliableSequencesMatches || (Bunch.ChSequence == InPartialBunch->ChSequence);

	// Unreliable partial bunches use the packet sequence, and since we can merge multiple bunches into a single packet,
	// it's perfectly legal for the ChSequence to match in this case.
	// Reliable partial bunches must be in consecutive order though
	bSequenceMatches = InPartialBunch->bReliable ? bReliableSequencesMatches : bUnreliableSequenceMatches;
}

if ( InPartialBunch && !InPartialBunch->bPartialFinal && bSequenceMatches && InPartialBunch->bReliable == Bunch.bReliable )
{
	// Merge.
	UE_LOG(LogNetPartialBunch, Verbose, TEXT("Merging Partial Bunch: %d Bytes"), Bunch.GetBytesLeft() );

	if ( !Bunch.bHasPackageMapExports && Bunch.GetBitsLeft() > 0 )
	{
		InPartialBunch->AppendDataFromChecked( Bunch.GetDataPosChecked(), Bunch.GetBitsLeft() );
	}

	// Only the final partial bunch should ever be non byte aligned. This is enforced during partial bunch creation
	// This is to ensure fast copies/appending of partial bunches. The final partial bunch may be non byte aligned.
	if (!Bunch.bHasPackageMapExports && !Bunch.bPartialFinal && (Bunch.GetBitsLeft() % 8 != 0))
	{
		UE_LOG(LogNetPartialBunch, Warning, TEXT("Corrupt partial bunch. Non-final partial bunches are expected to be byte-aligned. bHasPackageMapExports = %d, bPartialFinal = %d, BitsLeft = %u. %s"),
			Bunch.bHasPackageMapExports ? 1 : 0, Bunch.bPartialFinal ? 1 : 0, Bunch.GetBitsLeft(), *Describe());
		Bunch.SetError();
		return false;
	}

	// Advance the sequence of the current partial bunch so we know what to expect next
	InPartialBunch->ChSequence = Bunch.ChSequence;

	if (Bunch.bPartialFinal)
	{
		LogPartialBunch(TEXT("Completed Partial Bunch."), Bunch, *InPartialBunch);

		if ( Bunch.bHasPackageMapExports )
		{
			// Shouldn't have these, they only go in initial partial export bunches
			UE_LOG(LogNetPartialBunch, Warning, TEXT("Corrupt partial bunch. Final partial bunch has package map exports. %s"), *Describe());
			Bunch.SetError();
			return false;
		}

		HandleBunch = InPartialBunch;

		InPartialBunch->bPartialFinal			= true;
		InPartialBunch->bClose					= Bunch.bClose;
		InPartialBunch->CloseReason				= Bunch.CloseReason;
		InPartialBunch->bIsReplicationPaused	= Bunch.bIsReplicationPaused;
		InPartialBunch->bHasMustBeMappedGUIDs	= Bunch.bHasMustBeMappedGUIDs;
	}
	else
	{
		LogPartialBunch(TEXT("Received Partial Bunch."), Bunch, *InPartialBunch);
	}
}
else
{
	// 一些异常情况的错误处理

	if (InPartialBunch)
	{
		delete InPartialBunch;
		InPartialBunch = NULL;
	}	
}

这里的序列号一致有两个分支判断：当bReliable为true的时候，新Bunch的序列号必须比InPartialBunch的序列号大1；当bReliable为false的时候要求新Bunch的序列号与InPartialBunch的序列号差值要小于等于1。这里允许非可靠包的不同PartialBunch序列号相等的情况，估计是在执行包拆分的时候最后一个包的大小比较小，导致在拼装Packet的时候与前面的包合并到一个Packet里了。

拼接完成数据之后，检查当前Bunch是否是分包里的最后一个子包，如果是则代表一个完整包的多个分包都接收完毕了，可以将HandleBunch设置为当前的InPartialBunch。在获取了一个有效的HandleBunch之后，再调用ReceivedSequencedBunch来处理这个完整的消息包。

bool UChannel::ReceivedNextBunch( FInBunch & Bunch, bool & bOutSkipAck )
{
	FInBunch* HandleBunch = &Bunch;
	if (Bunch.bPartial)
	{
		//省略大包的合并相关代码
	}
	if ( HandleBunch != NULL )
	{
		const bool bBothSidesCanOpen = Connection->Driver && Connection->Driver->ChannelDefinitionMap[ChName].bServerOpen && Connection->Driver->ChannelDefinitionMap[ChName].bClientOpen;

		if ( HandleBunch->bOpen )
		{
			// 忽略一些channel打开时的逻辑处理
		}

		if ( !bBothSidesCanOpen )	// Voice channels can open from both side simultaneously, so ignore this logic until we resolve this
		{
			// 忽略一些channel打开时的逻辑处理
		}

		// Receive it in sequence.
		return ReceivedSequencedBunch( *HandleBunch );
	}

	return false;
}

ReceivedSequencedBunch只是简单的进行Bunch转发，并处理一下Channel关闭的逻辑。这里的UChannel::ReceivedBunch是一个纯虚函数，具体的Bunch内数据解析逻辑依赖于三个子类的重载,这里我们将不再跟进,UActorChannel::ReceivedBunch的相关内容将在后续的Actor同步中进行介绍:

bool UChannel::ReceivedSequencedBunch( FInBunch& Bunch )
{
	SCOPED_NAMED_EVENT(UChannel_ReceivedSequencedBunch, FColor::Green);
	// Handle a regular bunch.
	if ( !Closing )
	{
		ReceivedBunch( Bunch );
	}

	// We have fully received the bunch, so process it.
	if( Bunch.bClose )
	{
		// 忽略channel关闭的处理
		return true;
	}
	return false;
}
/** Handle an incoming bunch. */
virtual void UChannel::ReceivedBunch( FInBunch& Bunch ) PURE_VIRTUAL(UChannel::ReceivedBunch,);

void UActorChannel::ReceivedBunch( FInBunch & Bunch );
void UVoiceChannel::ReceivedBunch(FInBunch& Bunch);
void UControlChannel::ReceivedBunch( FInBunch& Bunch );

可靠传输

众所周知UDP是一个不可靠的协议，在其协议规范上并没有做类似于TCP的可靠收发保证。因此所有使用UDP作为底层通信协议的业务系统都需要在UDP的基础上模拟一个类似于TCP的可靠传输协议出来，UE也不例外。为了实现可靠传输，首先需要在每个Packet上赋予一个递增序列号，作为这个Packet的唯一标识符。并且通信的双方都需要对已经收到的包进行确认，也就是常说的ACK。为了方便的维护这个双端包序列号的发送与确认，UE里专门设计了一个结构FNetPacketNotify，在这个结构体里有多个序列号相关的成员变量:

/** 
	FNetPacketNotify - Drives delivery of sequence numbers, acknowledgments and notifications of delivery sequence numbers
*/
class FNetPacketNotify
{
private:
	// Track incoming sequence data
	SequenceHistoryT InSeqHistory;		// BitBuffer containing a bitfield describing the history of received packets
	SequenceNumberT InSeq;				// Last sequence number received and accepted from remote
	SequenceNumberT InAckSeq;			// Last sequence number received from remote that we have acknowledged, this is needed since we support accepting a packet but explicitly not acknowledge it as received.
	SequenceNumberT InAckSeqAck;		// Last sequence number received from remote that we have acknowledged and also knows that the remote has received the ack, used to calculate how big our history must be

	// Track outgoing sequence data
	SequenceNumberT OutSeq;				// Outgoing sequence number
	SequenceNumberT OutAckSeq;			// Last sequence number that we know that the remote side have received.
};

InSeq和OutSeq的意义很明显，分别代表接收到的最大包序号和下一个发出包的序号。OutAckSeq代表已发出的包中收到的对方确认了的最大包序号，对应的InAckSeq代表已接收到的包中给对端发送ACK的最大包序号。由于UE并不是收到一个对端的包之后就立即对这个包进行ACK，所以这个InAckSeq并不是总等于InSeq。这些字段的初始值会在连接建立的时候予以初始化:

void UNetConnection::InitSequence(int32 IncomingSequence, int32 OutgoingSequence)
{
	// Make sure the sequence hasn't already been initialized on the server, and ignore multiple initializations on the client
	check(InPacketId == -1 || Driver->ServerConnection != nullptr);

	if (InPacketId == -1 && CVarRandomizeSequence.GetValueOnAnyThread() > 0)
	{
		// Initialize the base UNetConnection packet sequence (not very useful/effective at preventing attacks)
		InPacketId = IncomingSequence - 1;
		OutPacketId = OutgoingSequence;
		OutAckPacketId = OutgoingSequence - 1;
		LastNotifiedPacketId = OutAckPacketId;

		// Initialize the reliable packet sequence (more useful/effective at preventing attacks)
		InitInReliable = IncomingSequence & (MAX_CHSEQUENCE - 1);
		InitOutReliable = OutgoingSequence & (MAX_CHSEQUENCE - 1);

		InReliable.Init(InitInReliable, InReliable.Num());
		OutReliable.Init(InitOutReliable, OutReliable.Num());

		PacketNotify.Init(InPacketId, OutPacketId);

		UE_LOG(LogNet, Verbose, TEXT("InitSequence: IncomingSequence: %i, OutgoingSequence: %i, InitInReliable: %i, InitOutReliable: %i"), IncomingSequence, OutgoingSequence, InitInReliable, InitOutReliable);
	}
}

void FNetPacketNotify::Init(SequenceNumberT InitialInSeq, SequenceNumberT InitialOutSeq)
{
	InSeqHistory.Reset();
	InSeq = InitialInSeq;
	InAckSeq = InitialInSeq;
	InAckSeqAck = InitialInSeq;
	OutSeq = InitialOutSeq;
	OutAckSeq = SequenceNumberT(InitialOutSeq.Get() - 1);
}

最后的InAckSeqAck则比较拗口，代表对端确认的自己这边接收到的包的最大值。举个例子来说说一下这些字段的更新逻辑：

1. A与B在连接建立的时候会在握手信息里商定双方的Seq初始值，假设A的初始Seq为100,同时B的初始Seq为200，那么A.InSeq=199, A.InAckSeq=199, A.InAckSeqAck=199, A.OutSeq=100, A.OutAckSeq=99, B.InSeq=99, B.InAckSeq=99, B.InAckSeqAck=99, B.OutSeq=200, B.OutAckSeq=199。
2. A往B发送了一个序号为100的包，此时更新A.OutSeq=100,同时携带对B发出的199包的ACK信息，更新A.InAckSeq=199。
3. 当B收到之后，更新B.InSeq=100，B.OutAckSeq=199，此时不急于发送100这个ACK, 等待后续发包的时候带上这个ACK。
4. 后面B在往A发送一个编号为200的包的时候，顺带的加上对A发出的100包的ACK信息，此时B.InAckSeq=100, B.OutSeq=200。
5. 在A接收到B发出的200包之后，更新A.InSeq=200，同时A知道对端B已经接收到了A发出的100包，此时更新A.OutAckSeq=100。
6. 接下来A发送给B一个编号为101的包，更新A.OutSeq=101，同时带上对B发出的200包的ACK信息，更新A.InAckSeq=200。
7. 当B收到这个101的包之后，更新B.InSeq=101，同时从这个包里解析出来包含对于B发出的200包的ACK信息，因此更新B.OutAckSeq=200。
8. 由于B发送的200包里携带了针对于A发送的100包的ACK,所以这个时候B知道A已经确认了B已经接收了100包这个信息，所以此时可以更新B.InAckSeqAck=100。

在UE的可靠UDP设计里，消息传输的有序性并不是建立在Packet的按照顺序接收的基础上，而是建立在可靠Bunch的按序处理的基础上，因为可靠传输的Bunch会赋予一个连续递增的序列号。当UE接收到一个Packet之后，会解析出这个Packet里所携带的所有Bunch,然后根据这个Packet的可靠性来做后续的处理：

1. 如果当前Packet是不需要可靠传输的，那么这些Bunch就会直接发送到逻辑层去处理
2. 如果当前Packet是需要可靠传输的，那么这些Bunch会投递到所属的Channel里进行排序，当出现连续Bunch的时候才会被分发到逻辑层

在这样的设计下，UE里每次ACK都代表一个独立的包被接收，而不是与TCP一样代表所有序号小于等于这个ACK的包被接收。当一个包长时间没有接收到对应的ACK的时候，就可以当作这个包的数据已经丢失。如果这个包是一个不可靠包，那么本地将不做任何处理；如果这个包是一个可靠包，那么UE会将这个包里的所有Bunch都解析出来然后重新放到Bunch的发送队列里等待后续组成Packet重新发送过去。举个例子来说，如果A向B发出了四个包：101不可靠， 102不可靠, 103可靠，包含Bunch(10)， 104可靠包含Bunch(11)。但是B只接收到了101和104，那么会按照包的上升序来分别处理101,104，但是处理104的时候发现内部的可靠Bunch的序列号11无法与之前接收到的最大可靠Bunch的序列号9连接起来，因此会先将Bunch(11)暂时存下来。后续B往A发包的时候会带上针对101,104的ACK。A收到B发出的101,104的ACK之后，知道了102,103两个包丢失了，由于102是不可靠包所以不会去重传这个包，但是103是可靠包，所以会将其内部的Bunch(10)拿出来放到发送Bunch队列的头部。在后面A会构造一个新的编号为105的可靠包，包里的内容是Bunch(10),Bunch(12)。当B收到这个105包之后，将Bunch(10),Bunch(12)与暂存了的Bunch(11)合并起来排序，然后不断的获取其内部可以在Bunch(9)之后按序接收的所有包，并执行逻辑层的分发。

上述内容就是UE可靠UDP的大致实现框架，接下来再来讲解一下具体的实现细节。首先是Packet的序列号分配，这个分配接口为FNetPacketNotify::CommitAndIncrementOutSeq,调用者为前面介绍过的FlushNet:

FNetPacketNotify::SequenceNumberT FNetPacketNotify::CommitAndIncrementOutSeq()
{
   // we have not written a header...this is a fail.
   check(WrittenHistoryWordCount != 0);

   // Add entry to the ack-record so that we can update the InAckSeqAck when we received the ack for this OutSeq.
   AckRecord.Enqueue( {OutSeq, WrittenInAckSeq} );
   WrittenHistoryWordCount = 0u;
   
   return ++OutSeq;
}

在这个函数里除了对OutSeq进行自增之外，还往AckRecord这个队列里添加了一个元素。这个AckRecord队列的作用是记录每次发出的包序列号以及这个包附带的ACK序列号:

struct FSentAckData
{
	SequenceNumberT OutSeq;	// 发送出去的包的序列号
	SequenceNumberT InAckSeq; // 这个包所携带的ACK序列号
};
typedef TResizableCircularQueue<FSentAckData, TInlineAllocator<128>> AckRecordT;

AckRecordT AckRecord;				// Track acked seq for each sent packet to track size of ack history

这样当自己接收到对端发送过来的一个针对己方发送的包的ACK时，就可以从这个AckRecord里找到自己发送的这个包所携带的针对对端的ACK信息，从而就可以更新InAckSeqAck字段了，这个查找包序号对应的ACK序号的接口为FNetPacketNotify::UpdateInAckSeqAck:

FNetPacketNotify::SequenceNumberT FNetPacketNotify::UpdateInAckSeqAck(SequenceNumberT::DifferenceT AckCount, SequenceNumberT AckedSeq)
{
	if ((SIZE_T)AckCount <= AckRecord.Count())
	{
		if (AckCount > 1)
		{
			AckRecord.PopNoCheck(AckCount - 1);
		}

		FSentAckData AckData = AckRecord.PeekNoCheck();
		AckRecord.PopNoCheck();

		// verify that we have a matching sequence number
		if (AckData.OutSeq == AckedSeq)
		{
			return AckData.InAckSeq;
		}
	}

	// Pessimistic view, should never occur but we do want to know about it if it would
	ensureMsgf(false, TEXT("FNetPacketNotify::UpdateInAckSeqAck - Failed to find matching AckRecord for %u"), AckedSeq.Get());
	
	return SequenceNumberT(AckedSeq.Get() - MaxSequenceHistoryLength);
}

有了包序列号之后，就需要在发送Packet的时候将包序号和ACK序号都附加到Packet里，这个添加序列号的逻辑在前面介绍的WritePacketHeader里:

// IMPORTANT:
// WritePacketHeader must ALWAYS write the exact same number of bits as we go back and rewrite the header
// right before we put the packet on the wire.
void UNetConnection::WritePacketHeader(FBitWriter& Writer)
{
	// If this is a header refresh, we only serialize the updated serial number information
	const bool bIsHeaderUpdate = Writer.GetNumBits() > 0u;

	// Header is always written first in the packet
	FBitWriterMark Reset;
	FBitWriterMark Restore(Writer);
	Reset.PopWithoutClear(Writer);
	
	// Write notification header or refresh the header if used space is the same.
	bool bWroteHeader = PacketNotify.WriteHeader(Writer, bIsHeaderUpdate);

#if !UE_BUILD_SHIPPING
	checkf(Writer.GetNumBits() <= MAX_PACKET_RELIABLE_SEQUENCE_HEADER_BITS, TEXT("WritePacketHeader exceeded the max allowed bits. Wrote %d. Max %d"), Writer.GetNumBits(), MAX_PACKET_RELIABLE_SEQUENCE_HEADER_BITS);
#endif

	// 忽略一些代码
}


// These methods must always write and read the exact same number of bits, that is the reason for not using WriteInt/WrittedWrappedInt
bool FNetPacketNotify::WriteHeader(FBitWriter& Writer, bool bRefresh)
{
	// we always write at least 1 word
	SIZE_T CurrentHistoryWordCount = FMath::Clamp<SIZE_T>((GetCurrentSequenceHistoryLength() + SequenceHistoryT::BitsPerWord - 1u) / SequenceHistoryT::BitsPerWord, 1u, SequenceHistoryT::WordCount);

	// We can only do a refresh if we do not need more space for the history
	if (bRefresh && (CurrentHistoryWordCount > WrittenHistoryWordCount))
	{
		return false;
	}

	// How many words of ack data should we write? If this is a refresh we must write the same size as the original header
	WrittenHistoryWordCount = bRefresh ? WrittenHistoryWordCount : CurrentHistoryWordCount;
	// This is the last InAck we have acknowledged at this time
	WrittenInAckSeq = InAckSeq;

	SequenceNumberT::SequenceT Seq = OutSeq.Get();
	SequenceNumberT::SequenceT AckedSeq = InAckSeq.Get();

	// Pack data into a uint
	uint32 PackedHeader = FPackedHeader::Pack(Seq, AckedSeq, WrittenHistoryWordCount - 1);

	// Write packed header
	Writer << PackedHeader;

	// Write ack history
	InSeqHistory.Write(Writer, WrittenHistoryWordCount);

	UE_LOG_PACKET_NOTIFY(TEXT("FNetPacketNotify::WriteHeader - Seq %u, AckedSeq %u bReFresh %u HistorySizeInWords %u"), Seq, AckedSeq, bRefresh ? 1u : 0u, WrittenHistoryWordCount);

	return true;
}

在这个WriteHeader函数里会在头部先写入一个uint32，这个uint32是由三个分量组成的:Seq代表当前Packet的序列号，占据14bit，AckedSeq代表已经收到包的最大序列号，占据14bit, 还有一个占据4bit的WrittenHistoryWordCount。三个分量刚好组合成32bit:

struct FPackedHeader
{
	using SequenceNumberT = FNetPacketNotify::SequenceNumberT;

	static_assert(FNetPacketNotify::SequenceNumberBits <= 14, "SequenceNumbers must be smaller than 14 bits to fit history word count");

	enum { HistoryWordCountBits = 4 };
	enum { SeqMask				= (1 << FNetPacketNotify::SequenceNumberBits) - 1 };
	enum { HistoryWordCountMask	= (1 << HistoryWordCountBits) - 1 };
	enum { AckSeqShift			= HistoryWordCountBits };
	enum { SeqShift				= AckSeqShift + FNetPacketNotify::SequenceNumberBits };
	
	static uint32 Pack(SequenceNumberT Seq, SequenceNumberT AckedSeq, SIZE_T HistoryWordCount)
	{
		uint32 Packed = 0u;

		Packed |= Seq.Get() << SeqShift;
		Packed |= AckedSeq.Get() << AckSeqShift;
		Packed |= HistoryWordCount & HistoryWordCountMask;

		return Packed;
	}

	static SequenceNumberT GetSeq(uint32 Packed) { return SequenceNumberT(Packed >> SeqShift & SeqMask); }
	static SequenceNumberT GetAckedSeq(uint32 Packed) { return SequenceNumberT(Packed >> AckSeqShift & SeqMask); }
	static SIZE_T GetHistoryWordCount(uint32 Packed) { return (Packed & HistoryWordCountMask); }
};

但是这样的设计下，序列号只有14bit可用，最大值只有16383，正常通信情况下这个最大值几分钟之内就会被超过。UE在处理序列号溢出的时候就是简单的进行回环，即序列号16383的下一个是0。为了方便的支持基于回环的加减与比较，UE为包序列号专门设计了一个类型SequenceNumberT:

/** Helper class to work with sequence numbers */
template <SIZE_T NumBits, typename SequenceType>
class TSequenceNumber
{
	static_assert(TIsSigned<SequenceType>::Value == false, "The base type for sequence numbers must be unsigned");

public:
	using SequenceT = SequenceType;
	using DifferenceT = int32;

	// Constants
	enum { SeqNumberBits = NumBits };
	enum { SeqNumberCount = SequenceT(1) << NumBits };
	enum { SeqNumberHalf = SequenceT(1) << (NumBits - 1) };
	enum { SeqNumberMax = SeqNumberCount - 1u };
	enum { SeqNumberMask = SeqNumberMax };

	/** Default constructor */
	TSequenceNumber() : Value(0u) {}

	/** Constructor with given value */
	TSequenceNumber(SequenceT ValueIn) : Value(ValueIn & SeqNumberMask) {}
	
	/** Get Current Value */	
	SequenceT Get() const { return Value; }

	/** Diff between sequence numbers (A - B) only valid if (A - B) < SeqNumberHalf */
	static DifferenceT Diff(TSequenceNumber A, TSequenceNumber B);
	
	/** return true if this is > Other, this is only considered to be the case if (A - B) < SeqNumberHalf since we have to be able to detect wraparounds */
	bool operator>(const TSequenceNumber& Other) const { return (Value != Other.Value) && (((Value - Other.Value) & SeqNumberMask) < SeqNumberHalf); }

	/** Check if this is >= Other, See above */
	bool operator>=(const TSequenceNumber& Other) const { return ((Value - Other.Value) & SeqNumberMask) < SeqNumberHalf; }

	/** Pre-increment and wrap around */
	TSequenceNumber& operator++() { Increment(1u); return *this; }
	
	/** Post-increment and wrap around */
	TSequenceNumber operator++(int) { TSequenceNumber Tmp(*this); Increment(1u); return Tmp; }

private:
	void Increment(SequenceT InValue) { *this = TSequenceNumber(Value + InValue); }
	SequenceT Value;
};



enum { SequenceNumberBits = 14 };
enum { MaxSequenceHistoryLength = 256 };

typedef TSequenceNumber<SequenceNumberBits, uint16> SequenceNumberT;

这里执行比较的逻辑比较绕，简单来说就是A与B之间基于回环的差值小于8192的时候才认为A的序列号比B小，可以拆分为下面的几种情况来方便理解:

1. 0<int(A) - int(B)<8191，则认为A比B大, 样例A=2, B=1
2. int(A) - int(B) >=8191,则认为B比A大，样例A=16000, B=1
3. -8192<int(A) - int(B) <0，则认为A比B小，样例A=1, B=4000
4. int(A) - int(B) < -8192, 则认为B比A大, 样例A=1, B=14000

知道了序列号的大小比较规则之后，计算序列号的差值的逻辑就更好理解了:

template <SIZE_T NumBits, typename SequenceType>
typename TSequenceNumber<NumBits, SequenceType>::DifferenceT TSequenceNumber<NumBits, SequenceType>::Diff(TSequenceNumber A, TSequenceNumber B) 
{ 
	constexpr SIZE_T ShiftValue = sizeof(DifferenceT)*8 - NumBits;

	const SequenceT ValueA = A.Value;
	const SequenceT ValueB = B.Value;

	return (DifferenceT)((ValueA - ValueB) << ShiftValue) >> ShiftValue;
};

举例来说上面这个函数的计算过程，假设A=10, B=250，则这两个uint16相减之后会得到65296，这个差值明显大于了我们指定的序列号上限16383，所以这里使用先左移后右移的方式来抹除多余的两个bit，并最终得到结果-240。

WrittenHistoryWordCount这个值用来表示在PackedHeader这个整数之后写入的动态数据的大小，只有4bit有效位，同时还被Clamp限制其取值范围为[1,8]。但是这个值的单位是typedef uint32 wordT，即写入的动态数据是以wordT来对齐的，每个wordT有四个字节，所以在后续写入的动态数据的大小上限为32byte，也就是256bit。

InSeqHistory.Write(Writer, WrittenHistoryWordCount)写入的动态数据代表当前ACK包序号AckedSeq之前的若干连续包的接收状态，如果一个包被接收了则对应的bit会被设置为1，如果一个包没有被接收那么对应的bit就会被设置为0。这个标记接口为FNetPacketNotify::AckSeq,第一个参数为要ACK的包编号，函数内部会通过循环来不断的更新InAckSeq，所有中间的包对应的bit都会设置为0，代表丢失，最后一个包的接收状态由传入的参数IsAck来控制。在FNetPacketNotify提供了这个AckSeq的封装， 单参数的AckSeq代表确认了这个包的接收，NakSeq代表确认了这个包的丢失：

void FNetPacketNotify::AckSeq(SequenceNumberT AckedSeq, bool IsAck)
{
	check( AckedSeq == InSeq);

	while (AckedSeq > InAckSeq)
	{
		++InAckSeq;

		const bool bReportAcked = InAckSeq == AckedSeq ? IsAck : false;

		UE_LOG_PACKET_NOTIFY(TEXT("FNetPacketNotify::AckSeq - AckedSeq: %u, IsAck %u"), InAckSeq.Get(), bReportAcked ? 1u : 0u);

		InSeqHistory.AddDeliveryStatus(bReportAcked);		
	}
}

/** Mark Seq as received and update current InSeq, missing sequence numbers will be marked as lost */
void AckSeq(SequenceNumberT Seq) { AckSeq(Seq, true); }

/** Explicitly mark Seq as not received and update current InSeq, additional missing sequence numbers will be marked as lost */
void NakSeq(SequenceNumberT Seq) { AckSeq(Seq, false); }

由于动态数据最多只有256bit，只能覆盖当前最新包的前256个包的接收状态，如果一个比较久远的包没有被接收到，那么这里就会出现问题，所以在WriterHeader的开头会有这个长度的检查，发现异常则直接返回false:

bool FNetPacketNotify::WriteHeader(FBitWriter& Writer, bool bRefresh)
{
	// we always write at least 1 word
	SIZE_T CurrentHistoryWordCount = FMath::Clamp<SIZE_T>((GetCurrentSequenceHistoryLength() + SequenceHistoryT::BitsPerWord - 1u) / SequenceHistoryT::BitsPerWord, 1u, SequenceHistoryT::WordCount);

	// We can only do a refresh if we do not need more space for the history
	if (bRefresh && (CurrentHistoryWordCount > WrittenHistoryWordCount))
	{
		return false;
	}
	// 省略后续代码
}

这里之所以写入这个InSeqHistory信息到Packet里是为了单个Packet对多个已经接收的包来执行ACK，如果每次接收到一个包都单独发出ACK消息的话，会非常浪费流量。因此这里使用bit来代表一个包的接收状态，可以非常的高效的对多个包执行批量ACK，节省了很多流量。

虽然InSeqHistory可以最大容纳256bit，但是每次都完整的写入当前确认包之前的256个包的接收状态有点过滤浪费流量了。所以这里会使用GetCurrentSequenceHistoryLength来判断有多少个包的状态需要写入，如果某个包A之前的包接收状态对方已经知道了，那么此时只需要将A之后到InAckSeq之间的包的接收状态进行发送，也就是(InAckSeqAck,InAckSeq)这个区间的包，这里就是InAckSeqAck这个变量的唯一作用，计算哪些包的接收状态对方还不知道:

SIZE_T FNetPacketNotify::GetCurrentSequenceHistoryLength() const
{
	if (InAckSeq >= InAckSeqAck)
	{
		return (SIZE_T)SequenceNumberT::Diff(InAckSeq, InAckSeqAck);
	}
	else
	{
		// Worst case send full history
		return SequenceHistoryT::Size;
	}
}

当网络通信的对端接收到这个Packet的时候，在UNetConnection::ReceivedPacket也需要按照WriterHeader的格式来解析头部的数据:

bool FNetPacketNotify::ReadHeader(FNotificationHeader& Data, FBitReader& Reader) const
{
	// Read packed header
	uint32 PackedHeader = 0;	
	Reader << PackedHeader;

	// unpack
	Data.Seq = FPackedHeader::GetSeq(PackedHeader);
	Data.AckedSeq = FPackedHeader::GetAckedSeq(PackedHeader);
	Data.HistoryWordCount = FPackedHeader::GetHistoryWordCount(PackedHeader) + 1;

	// Read ack history
	Data.History.Read(Reader, Data.HistoryWordCount);

	UE_LOG_PACKET_NOTIFY(TEXT("FNetPacketNotify::ReadHeader - Seq %u, AckedSeq %u HistorySizeInWords %u"), Data.Seq.Get(), Data.AckedSeq.Get(), Data.HistoryWordCount);

	return Reader.IsError() == false;
}

void UNetConnection::ReceivedPacket( FBitReader& Reader, bool bIsReinjectedPacket)
{
	SCOPED_NAMED_EVENT(UNetConnection_ReceivedPacket, FColor::Green);
	AssertValid();

	// Handle PacketId.
	if( Reader.IsError() )
	{
		ensureMsgf(false, TEXT("Packet too small") );
		return;
	}

	// 省略一些代码

	FChannelsToClose ChannelsToClose;

	if (IsInternalAck())
	{
		++InPacketId;
	}	
	else
	{
		// Read packet header
		FNetPacketNotify::FNotificationHeader Header;
		if (!PacketNotify.ReadHeader(Header, Reader))
		{
			CLOSE_CONNECTION_DUE_TO_SECURITY_VIOLATION(this, ESecurityEvent::Malformed_Packet, TEXT("Failed to read PacketHeader"));
			return;
		}
		// 省略后续代码
	}
}

当解析完这个Header信息之后，就可以知道当前新Packet的包序列号了，此时需要将这个新包序列号与之前记录的收到的最大包序列号做一个比较。由于当前UE的接收设计，只有新包序列号大于之前记录的InSeq的时候才会去处理，否则会直接认为这个包是无效包。所以这个ReceivedPacket的开头会首先计算这个序列号的差值:

/**
 * Gets the delta between the present sequence, and the sequence inside the specified header - if the delta is positive
 */
SequenceNumberT::DifferenceT GetSequenceDelta(const FNotificationHeader& NotificationData)
{
	if (NotificationData.Seq > InSeq && NotificationData.AckedSeq >= OutAckSeq && OutSeq > NotificationData.AckedSeq)
	{
		return SequenceNumberT::Diff(NotificationData.Seq, InSeq);
	}
	else
	{
		return 0;
	}
}

void UNetConnection::ReceivedPacket( FBitReader& Reader, bool bIsReinjectedPacket)
{
	// 省略前述代码
	const int32 PacketSequenceDelta = PacketNotify.GetSequenceDelta(Header);
	if(PacketSequenceDelta > 0)
	{
		// 暂时省略一些代码
	}
	else
	{
		// 暂时省略一些代码
	}
	// 省略后续代码
}

如果比较出来的差值小于0，则代表出现了历史消息的重复接收，因此可以直接忽略掉。但是这里还做了一点额外工作，如果这种历史消息出现了多次，超过了指定的阈值CVarNetPacketOrderCorrectionEnableThreshold, 那么可以认为当前的网络环境不怎么好，此时会创建一个乱序接收包的队列PacketOrderCache:

static TAutoConsoleVariable<int32> CVarNetDoPacketOrderCorrection(TEXT("net.DoPacketOrderCorrection"), 1,
	TEXT("Whether or not to try to fix 'out of order' packet sequences, by caching packets and waiting for the missing sequence."));

static TAutoConsoleVariable<int32> CVarNetPacketOrderCorrectionEnableThreshold(TEXT("net.PacketOrderCorrectionEnableThreshold"), 1,
	TEXT("The number of 'out of order' packet sequences that need to occur, before correction is enabled."));

static TAutoConsoleVariable<int32> CVarNetPacketOrderMaxCachedPackets(TEXT("net.PacketOrderMaxCachedPackets"), 32,
	TEXT("(NOTE: Must be power of 2!) The maximum number of packets to cache while waiting for missing packet sequences, before treating missing packets as lost."));

if(PacketSequenceDelta > 0)
{
	// 省略一些代码
}
else
{
	TotalOutOfOrderPackets++;
	Driver->InOutOfOrderPackets++;

	if (!PacketOrderCache.IsSet() && CVarNetDoPacketOrderCorrection.GetValueOnAnyThread() != 0)
	{
		int32 EnableThreshold = CVarNetPacketOrderCorrectionEnableThreshold.GetValueOnAnyThread();

		if (TotalOutOfOrderPackets >= EnableThreshold)
		{
			UE_LOG(LogNet, Verbose, TEXT("Hit threshold of %i 'out of order' packet sequences. Enabling out of order packet correction."), EnableThreshold);

			int32 CacheSize = FMath::RoundUpToPowerOfTwo(CVarNetPacketOrderMaxCachedPackets.GetValueOnAnyThread());

			PacketOrderCache.Emplace(CacheSize);
		}
	}

	// Protect against replay attacks
	// We already protect against this for reliable bunches, and unreliable properties
	// The only bunch we would process would be unreliable RPC's, which could allow for replay attacks
	// So rather than add individual protection for unreliable RPC's as well, just kill it at the source, 
	// which protects everything in one fell swoop
	return;
}

这里会对PacketOrderCache成员变量做容量初始化，这个成员是一个循环队列的Optional，默认情况下是空值，只有在上面出现重复包的时候才会去初始化:

/** Buffer of partially read (post-PacketHandler) sequenced packets, which are waiting for a missing packet/sequence */
TOptional<TCircularBuffer<TUniquePtr<FBitReader>>> PacketOrderCache;

这个循环队列里存储的数据是TUniquePtr<FBitReader>，其实就是一个Packet的数据，代表一个乱序接收的包，所以这个成员变量主要是作为临时buffer来将乱序接收到的包重整为有序包使用。了解了这个成员变量的作用之后，再去回顾一下之前计算的PacketSequenceDelta>0的分支处理,代码的开头会去初始化一些变量:

static TAutoConsoleVariable<int32> CVarNetPacketOrderMaxMissingPackets(TEXT("net.PacketOrderMaxMissingPackets"), 3,
	TEXT("The maximum number of missed packet sequences that is allowed, before treating missing packets as lost."));

const bool bPacketOrderCacheActive = !bFlushingPacketOrderCache && PacketOrderCache.IsSet();
const bool bCheckForMissingSequence = bPacketOrderCacheActive && PacketOrderCacheCount == 0;
const bool bFillingPacketOrderCache = bPacketOrderCacheActive && PacketOrderCacheCount > 0;
const int32 MaxMissingPackets = (bCheckForMissingSequence ? CVarNetPacketOrderMaxMissingPackets.GetValueOnAnyThread() : 0);

const int32 MissingPacketCount = PacketSequenceDelta - 1;

这里涉及到了太多的变量，需要先解释一下这些变量的用途:

1. bFlushingPacketOrderCache代表是否正在清空PacketOrderCache的连续包，目前我们可以把这个变量当作false
2. PacketOrderCacheCount代表PacketOrderCache的已占用槽位的个数
3. bPacketOrderCacheActive代表目前能否使用PacketOrderCache来缓存乱序包
4. bCheckForMissingSequence代表当前是否在检查丢失序列，成立条件为PacketOrderCache缓存激活且缓存为空
5. bFillingPacketOrderCache代表目前是否在检查序列填充，成立条件为PacketOrderCache缓存激活且缓存不为空
6. MaxMissingPackets代表最大能处理的中间丢失包的数量，默认为3
7. MissingPacketCount代表当前包与期望收到的下一个包的序列号差值，为0代表就是按序接收的下一个包

了解完这些变量的用途之后，才能去跟进后续的逻辑。开头就会有一个大的条件判断，其实就是排除掉顺序接收以及乱序差值太大的情况，如果乱序太大则会直接认为丢包等待对端重传所有包，不再使用这个PacketOrderCache来存住当前乱序包:

// Cache the packet if we are already caching, and begin caching if we just encountered a missing sequence, within range
if (bFillingPacketOrderCache || (bCheckForMissingSequence && MissingPacketCount > 0 && MissingPacketCount <= MaxMissingPackets))
{
	// 省略一些代码
	return;
}

if (MissingPacketCount > 10)
{
	UE_LOG(LogNetTraffic, Verbose, TEXT("High single frame packet loss. PacketsLost: %i %s" ), MissingPacketCount, *Describe());
}

InPacketsLost += MissingPacketCount;
InTotalPacketsLost += MissingPacketCount;
Driver->InPacketsLost += MissingPacketCount;
Driver->InTotalPacketsLost += MissingPacketCount;
InPacketId += PacketSequenceDelta;

PacketAnalytics.TrackInPacket(InPacketId, MissingPacketCount);

处理完这个if分支之后就会直接return，剩下的逻辑只会在MissingPacketCount=0 || MissingPacketCount > MaxMissingPackets的时候才会执行，只是做一些统计记录。

在这个if分支里，会执行PacketOrderCache的填充操作，这里会首先判断PacketOrderCache能否容下这个包，如果计算出来的LinearCacheIdx超限的话，则会校正到CacheCapacity - 1。

int32 LinearCacheIdx = PacketSequenceDelta - 1;
int32 CacheCapacity = PacketOrderCache->Capacity();
bool bLastCacheEntry = LinearCacheIdx >= (CacheCapacity - 1);

// The last cache entry is only set, when we've reached capacity or when we receive a sequence which is out of bounds of the cache
LinearCacheIdx = bLastCacheEntry ? (CacheCapacity - 1) : LinearCacheIdx;

这个PacketOrderCache循环数组的开头位置记录在PacketOrderCacheStartIdx，那么LinearCacheIdx偏移对应的位置就可以通过调用多次GetNextIndex计算出来:

int32 CircularCacheIdx = PacketOrderCacheStartIdx;

for (int32 LinearDec=LinearCacheIdx; LinearDec > 0; LinearDec--)
{
	CircularCacheIdx = PacketOrderCache->GetNextIndex(CircularCacheIdx);
}

在获取最终的数组索引CircularCacheIdx之后，如果这个位置目前暂时没有数据，则创建当前包的数据的一个UniquePtr塞入到对应的位置里，如果这个位置已经有数据，说明乱序包重复了，直接忽略:

TUniquePtr<FBitReader>& CurCachePacket = PacketOrderCache.GetValue()[CircularCacheIdx];

// Reset the reader to its initial position, and cache the packet
if (!CurCachePacket.IsValid())
{
	UE_LOG(LogNet, VeryVerbose, TEXT("'Out of Order' Packet Cache, caching sequence order '%i' (capacity: %i)"), LinearCacheIdx, CacheCapacity);

	CurCachePacket = MakeUnique<FBitReader>(Reader);
	PacketOrderCacheCount++;

	ResetReaderMark.Pop(*CurCachePacket);
}
else
{
	TotalOutOfOrderPackets++;
	Driver->InOutOfOrderPackets++;
}

一旦一个包进入了这个乱序缓存数组，这个包就不会被立即处理，而是等待调用FlushPacketOrderCache来处理若干个上升序的包,这个函数会在ReceivePacket后被立即调用，同时也会在PostTick里被调用。FlushPacketOrderCache会使用ReceivedPacket来重新走一遍这些包的接收流程:

void UNetConnection::ReceivedRawPacket( void* InData, int32 Count )
{
	// 省略很多代码
	if (Reader.GetBitsLeft() > 0)
	{
		ReceivedPacket(Reader);

		// Check if the out of order packet cache needs flushing
		FlushPacketOrderCache();
	}
}
void UNetConnection::PostTickDispatch()
{
	if (!IsInternalAck())
	{
#if DO_ENABLE_NET_TEST
		ReinjectDelayedPackets();
#endif

		FlushPacketOrderCache(/*bFlushWholeCache=*/true);
		PacketAnalytics.Tick();
	}
}
void UNetConnection::FlushPacketOrderCache(bool bFlushWholeCache/*=false*/)
{
	if (PacketOrderCache.IsSet() && PacketOrderCacheCount > 0)
	{
		TCircularBuffer<TUniquePtr<FBitReader>>& Cache = PacketOrderCache.GetValue();
		int32 CacheEndIdx = PacketOrderCache->GetPreviousIndex(PacketOrderCacheStartIdx);
		bool bEndOfCacheSet = Cache[CacheEndIdx].IsValid();

		bFlushingPacketOrderCache = true;

		// If the end of the cache has had its value set, this forces the flushing of the whole cache, no matter how many missing sequences there are.
		// The reason for this (other than making space in the cache), is that when we receive a sequence that is out of range of the cache,
		// it is stored at the end, and so the cache index no longer lines up with the sequence number - which it needs to.
		bFlushWholeCache = bFlushWholeCache || bEndOfCacheSet;

		while (PacketOrderCacheCount > 0)
		{
			TUniquePtr<FBitReader>& CurCachePacket = Cache[PacketOrderCacheStartIdx];

			if (CurCachePacket.IsValid())
			{
				UE_LOG(LogNet, VeryVerbose, TEXT("'Out of Order' Packet Cache, replaying packet with cache index: %i (bFlushWholeCache: %i)"), PacketOrderCacheStartIdx, (int32)bFlushWholeCache);

				ReceivedPacket(*CurCachePacket.Get());

				CurCachePacket.Reset();

				PacketOrderCacheCount--;
			}
			// Advance the cache only up to the first missing packet, unless flushing the whole cache
			else if (!bFlushWholeCache)
			{
				break;
			}

			PacketOrderCacheStartIdx = PacketOrderCache->GetNextIndex(PacketOrderCacheStartIdx);
		}

		bFlushingPacketOrderCache = false;
	}
}

注意这里即使包的序号不连续也是直接处理，即如果当前InSeq为10，且PacketOrderCache里的数据包编号为12, 14，那么12,14这两个包会被ReceivedPacket依次处理，而不是必须等待11, 13这两个包填充到PacketOrderCache。所以这里的PacketOrderCache的作用是对一些乱序接收到的包做一下临时缓存，避免已经接收到的包被丢弃。如果PacketOrderCache没有被启用，那么如果先接收到14，后接收到12，会导致14包先被处理，然后处理12包的时候发现是14包的以前的包，直接忽略，等待后续的重传。

如果是顺序接收MissingPacketCount=0的话，则不会进入这个大的条件判断，而是直接执行后续的处理:

// Lambda to dispatch delivery notifications, 
auto HandlePacketNotification = [&Header, &ChannelsToClose, this](FNetPacketNotify::SequenceNumberT AckedSequence, bool bDelivered)
{
	// Increase LastNotifiedPacketId, this is a full packet Id
	++LastNotifiedPacketId;
	++OutTotalNotifiedPackets;
	Driver->IncreaseOutTotalNotifiedPackets();

	// Sanity check
	if (FNetPacketNotify::SequenceNumberT(LastNotifiedPacketId) != AckedSequence)
	{
		CLOSE_CONNECTION_DUE_TO_SECURITY_VIOLATION(this, ESecurityEvent::Malformed_Packet, TEXT("LastNotifiedPacketId != AckedSequence"));
		return;
	}

	if (bDelivered)
	{
		ReceivedAck(LastNotifiedPacketId, ChannelsToClose);
	}
	else
	{
		ReceivedNak(LastNotifiedPacketId);
	};
};

// Update incoming sequence data and deliver packet notifications
// Packet is only accepted if both the incoming sequence number and incoming ack data are valid
PacketNotify.Update(Header, HandlePacketNotification);

这里的PacketNotify.Update会解析对方发送过来的包接收状态数组，并按照包序号来顺序调用上面的HandlePacketNotification，在HandlePacketNotification里会根据解析到的包是否被确认bDelivered来分别执行确认接收ReceiveAck还是确认丢失ReceiveNak:

template<class Functor>
FNetPacketNotify::SequenceNumberT::DifferenceT FNetPacketNotify::Update(const FNotificationHeader& NotificationData, Functor&& InFunc)
{
	const SequenceNumberT::DifferenceT InSeqDelta = GetSequenceDelta(NotificationData);

	if (InSeqDelta > 0)
	{
		UE_LOG_PACKET_NOTIFY(TEXT("FNetPacketNotify::Update - Seq %u, InSeq %u"), NotificationData.Seq.Get(), InSeq.Get());
	
		ProcessReceivedAcks(NotificationData, InFunc);

		// accept sequence
		InSeq = NotificationData.Seq;

		return InSeqDelta;
	}
	else
	{
		return 0;
	}
}

在FNetPacketNotify::Update就是真正的执行遍历传入的包接收状态数组的地方，在这个函数里首先根据传入的包确认序列号NotificationData.AckedSeq与之前记录的已经被确认收到的序列号OutAckSeq做比较，生成的差值AckCount会首先被UpdateInAckSeqAck用来更新InAckSeqAck为NotificationData.AckedSeq,这个UpdateInAckSeqAck其实就是从AckRecord数组里找到NotificationData.AckedSeq这个序号的包发送时顺带携带的ACK序列号，读者可以回顾一下UpdateInAckSeqAck的部分:

template<class Functor>
void FNetPacketNotify::ProcessReceivedAcks(const FNotificationHeader& NotificationData, Functor&& InFunc)
{
	if (NotificationData.AckedSeq > OutAckSeq)
	{
		UE_LOG_PACKET_NOTIFY(TEXT("Notification::ProcessReceivedAcks - AckedSeq: %u, OutAckSeq: %u"), NotificationData.AckedSeq.Get(), OutAckSeq.Get());

		SequenceNumberT::DifferenceT AckCount = SequenceNumberT::Diff(NotificationData.AckedSeq, OutAckSeq);

		// Update InAckSeqAck used to track the needed number of bits to transmit our ack history
		InAckSeqAck = UpdateInAckSeqAck(AckCount, NotificationData.AckedSeq);

		// ExpectedAck = OutAckSeq + 1
		SequenceNumberT CurrentAck(OutAckSeq);
		++CurrentAck;

		// Warn if the received sequence number is greater than our history buffer, since if that is the case we have to treat the data as lost.
		if (AckCount > (SequenceNumberT::DifferenceT)(SequenceHistoryT::Size))
		{
			UE_LOG_PACKET_NOTIFY_WARNING(TEXT("Notification::ProcessReceivedAcks - Missed Acks: AckedSeq: %u, OutAckSeq: %u, FirstMissingSeq: %u Count: %u"), NotificationData.AckedSeq.Get(), OutAckSeq.Get(), CurrentAck.Get(), AckCount - (SequenceNumberT::DifferenceT)(SequenceHistoryT::Size));
		}

		// Everything not found in the history buffer is treated as lost
		while (AckCount > (SequenceNumberT::DifferenceT)(SequenceHistoryT::Size))
		{
			--AckCount;
			InFunc(CurrentAck, false);
			++CurrentAck;
		}

		// For sequence numbers contained in the history we lookup the delivery status from the history
		while (AckCount > 0)
		{
			--AckCount;
			UE_LOG_PACKET_NOTIFY(TEXT("Notification::ProcessReceivedAcks Seq: %u - IsAck: %u HistoryIndex: %u"), CurrentAck.Get(), NotificationData.History.IsDelivered(AckCount) ? 1u : 0u, AckCount);
			InFunc(CurrentAck, NotificationData.History.IsDelivered(AckCount));
			++CurrentAck;
		}
		OutAckSeq = NotificationData.AckedSeq;
	}
}

接下来会从传入的历史包接收状态数组里NotificationData.History获取之前的多个包的接收状态，由于NotificationData.History这个数组最大只能容纳256个元素，所以对于太老的无法在NotificationData.History里记录的包，就直接认为已经丢失，因此这里会执行InFunc(CurrentAck, false)，这里的false就代表这个包丢失了。接下来就可以使用IsDelivered(AckCount)来获取CurrentAck对应包在NotificationData.History里记录的接收状态，并执行对应的InFunc。

对于汇报过来的丢失的Packet, 会遍历所有的Channel找到这个Packet里包含的所有Bunch，触发重新发送，此时SendRawBunch发送出去的Bunch会分配一个新的PacketId，不会使用之前的老PacketId:

//接收到另外一端反馈的丢包packetId
void UNetConnection::ReceivedNak( int32 NakPacketId )
{
	auto NakChannelFunc = [this](int32 NackedPacketId, uint32 ChannelIndex)
	{
		UChannel* const Channel = Channels[ChannelIndex];
		if (Channel)
		{
			Channel->ReceivedNak(NackedPacketId);
			if (Channel->OpenPacketId.InRange(NackedPacketId))
			{
				Channel->ReceivedAcks(); //warning: May destroy Channel.
			}
		}
	};

	// Invoke NakChannelFunc on all channels written for this PacketId
	FChannelRecordImpl::ConsumeChannelRecordsForPacket(ChannelRecord, NakPacketId, NakChannelFunc);

}

void UChannel::ReceivedNak( int32 NakPacketId )
{
	// 触发重传
	for( FOutBunch* Out=OutRec; Out; Out=Out->Next )
	{
		// Retransmit reliable bunches in the lost packet.
		if( Out->PacketId==NakPacketId && !Out->ReceivedAck )
		{
			check(Out->bReliable);
			UE_LOG(LogNetTraffic, Log, TEXT("      Channel %i nak); resending %i..."), Out->ChIndex, Out->ChSequence );
			Connection->SendRawBunch( *Out, 0 );
		}
	}
}

这里的OutRec就是之前在发送可靠Bunch的时候维护的可靠Bunch有序列表，这个列表会按照ChSequence的递增序号排列。注意这里的ChSequence的类型是int32，所以不需要考虑序列号的回环问题。但是在组包的时候为了降低要发送的数据流量，这里会只打包这个序列号对1024的取模结果，也就是说只写入这个int32的低10bit的数据:

enum { MAX_CHSEQUENCE = 1024 }; // Power of 2 >RELIABLE_BUFFER, covering loss/misorder time.
void FBitWriter::WriteIntWrapped(uint32 Value, uint32 ValueMax)
{
    // 直接按 LengthBits 写入 Value 的低位，不做 Value < ValueMax 的断言/修正
    const int32 LengthBits = FMath::CeilLogTwo(ValueMax);
    if (AllowAppend(LengthBits))
    {
        uint32 NewValue = 0;
        for (uint32 Mask=1; NewValue+Mask < ValueMax && Mask; Mask*=2, Num++)
        {
            if (Value & Mask)
            {
                Buffer[Num>>3] += GShift[Num&7];
                NewValue += Mask;
            }
        }
    }
}
int32 UNetConnection::SendRawBunch(FOutBunch& Bunch, bool InAllowMerge, const FNetTraceCollector* BunchCollector)
{
	// 省略很多代码
	if (Bunch.bReliable && !IsInternalAck())
	{
		SendBunchHeader.WriteIntWrapped(Bunch.ChSequence, MAX_CHSEQUENCE);
	}
}

对应的在接收到这个可靠Bunch的时候，会读取写入的10bit整数作为当前ChSequence的低10bit，然后根据当前已经顺序接收到的最大序列号InReliable[Bunch.ChIndex]来计算这个Bunch的ChSequence的完整32bit：

// Return the value of Max/2 <= Value-Reference+some_integer*Max < Max/2.
inline int32 BestSignedDifference( int32 Value, int32 Reference, int32 Max )
{
	return ((Value-Reference+Max/2) & (Max-1)) - Max/2;
}
inline int32 MakeRelative( int32 Value, int32 Reference, int32 Max )
{
	return Reference + BestSignedDifference(Value,Reference,Max);
}

void UNetConnection::ReceivedPacket( FBitReader& Reader, bool bIsReinjectedPacket)
{
	// 省略很多代码
	if ( Bunch.bReliable )
	{
		if ( IsInternalAck() )
		{
			// We can derive the sequence for 100% reliable connections
			Bunch.ChSequence = InReliable[Bunch.ChIndex] + 1;
		}
		else
		{
			// If this is a reliable bunch, use the last processed reliable sequence to read the new reliable sequence
			Bunch.ChSequence = MakeRelative( Reader.ReadInt( MAX_CHSEQUENCE ), InReliable[Bunch.ChIndex], MAX_CHSEQUENCE );
		}
	} 
}

对于确认接收到的Packet，会执行UNetConnection::ReceivedAck,这里会遍历所有的Channel，获取这个Packet里对应的所有Bunch，并将Bunch.ReceivedAck设置为1:

void UNetConnection::ReceivedAck(int32 AckPacketId, FChannelsToClose& OutChannelsToClose)
{
	UE_LOG(LogNetTraffic, Verbose, TEXT("   Received ack %i"), AckPacketId);

	auto AckChannelFunc = [this, &OutChannelsToClose](int32 AckedPacketId, uint32 ChannelIndex)
	{
		UChannel* const Channel = Channels[ChannelIndex];

		if (Channel)
		{
			if (Channel->OpenPacketId.Last == AckedPacketId) // Necessary for unreliable "bNetTemporary" channels.
			{
				Channel->OpenAcked = 1;
			}
				
			for (FOutBunch* OutBunch = Channel->OutRec; OutBunch; OutBunch = OutBunch->Next)
			{
				if (OutBunch->PacketId == AckedPacketId)
				{
					OutBunch->ReceivedAck = 1;
				}
			}
			Channel->ReceivedAck(AckedPacketId);
			EChannelCloseReason CloseReason;
			if (Channel->ReceivedAcks(CloseReason))
			{
				const FChannelCloseInfo Info = {ChannelIndex, CloseReason};
				OutChannelsToClose.Emplace(Info);
			}
		}
	};

	// Invoke AckChannelFunc on all channels written for this PacketId
	FChannelRecordImpl::ConsumeChannelRecordsForPacket(ChannelRecord, AckPacketId, AckChannelFunc);
}

void UChannel::ReceivedAck( int32 AckPacketId )
{
	// Do nothing. Most channels deal with this in Tick().
}

但是这里的ReceivedAck其实什么都没做，利用这个ReceivedAck字段的地方在后面调用的UChannel::ReceivedAcks里，这个函数会遍历这个OutRec链表里开头的所有已经标记为ReceivedAck==1的Bunch，执行delete来释放这个Bunch的内存，因为这个Bunch的数据已经被确认接收了，不会再触发重传，可以释放了:

bool UChannel::ReceivedAcks(EChannelCloseReason& OutCloseReason)
{
	check(Connection->Channels[ChIndex]==this);

	/*
	// Verify in sequence.
	for( FOutBunch* Out=OutRec; Out && Out->Next; Out=Out->Next )
		check(Out->Next->ChSequence>Out->ChSequence);
	*/

	// Release all acknowledged outgoing queued bunches.
	bool bCleanup = false;
	EChannelCloseReason CloseReason = EChannelCloseReason::Destroyed;
	
	while( OutRec && OutRec->ReceivedAck )
	{
		if (OutRec->bOpen)
		{
			// 忽略openbunch的处理
		}

		bCleanup = bCleanup || !!OutRec->bClose;

		if (OutRec->bClose)
		{
			CloseReason = OutRec->CloseReason;
		}

		FOutBunch* Release = OutRec;
		OutRec = OutRec->Next;
		delete Release;
		NumOutRec--;
	}

	// 忽略一些代码

	return false;
}

前面的内容都是在处理Packet里携带的ACK， 处理完ACK之后还需要处理Packet里的所有Bunch数据。之前在消息接收小结中介绍了Bunch接收逻辑在UChannel::ReceivedRawBunch里。由于之前我们已经介绍过顺序接收到Bunch的处理细节，因此这里只需要关心可靠Bunch的乱序接收处理逻辑:

void UChannel::ReceivedRawBunch( FInBunch & Bunch, bool & bOutSkipAck )
{
	// 省略一些代码
	if ( Bunch.bReliable && Bunch.ChSequence != Connection->InReliable[ChIndex] + 1 )
	{
		// We shouldn't hit this path on 100% reliable connections
		check( !Connection->IsInternalAck() );
		// If this bunch has a dependency on a previous unreceived bunch, buffer it.
		checkSlow(!Bunch.bOpen);

		// Verify that UConnection::ReceivedPacket has passed us a valid bunch.
		check(Bunch.ChSequence>Connection->InReliable[ChIndex]);

		// Find the place for this item, sorted in sequence.
		UE_LOG(LogNetTraffic, Log, TEXT("      Queuing bunch with unreceived dependency: %d / %d"), Bunch.ChSequence, Connection->InReliable[ChIndex]+1 );
		FInBunch** InPtr;
		for( InPtr=&InRec; *InPtr; InPtr=&(*InPtr)->Next )
		{
			if( Bunch.ChSequence==(*InPtr)->ChSequence )
			{
				// Already queued.
				return;
			}
			else if( Bunch.ChSequence<(*InPtr)->ChSequence )
			{
				// Stick before this one.
				break;
			}
		}

		FInBunch* New = new FInBunch(Bunch);
		New->Next     = *InPtr;
		*InPtr        = New;
		NumInRec++;

		if ( NumInRec >= RELIABLE_BUFFER )
		{
			Bunch.SetError();
			UE_LOG( LogNetTraffic, Error, TEXT( "UChannel::ReceivedRawBunch: Too many reliable messages queued up" ) );
			return;
		}

		checkSlow(NumInRec<=RELIABLE_BUFFER);
		//AssertInSequenced();
	}
	// 省略一些代码
}

这里的逻辑其实就是从InRec这个有序Bunch接收列表里插入当前的Bunch,使得InRec这个列表里的Bunch都按照ChSequence字段上升序排列。这里InRec里最多缓存255个Bunch数据，超过了这个限制则会报错。

后面如果接收到了一个可以处理的可靠Bunch，也就是Bunch.ChSequence == Connection->InReliable[ChIndex] + 1,那么在处理这个新Bunch的时候，会在ReceivedNextBunch更新Connection->InReliable[ChIndex]++，然后再使用while循环来遍历InRec这个缓存链表里所有与Connection->InReliable[ChIndex]连号的数据：

void UChannel::ReceivedRawBunch( FInBunch & Bunch, bool & bOutSkipAck )
{
	// 省略一些代码
	if ( Bunch.bReliable && Bunch.ChSequence != Connection->InReliable[ChIndex] + 1 )
	{
		// 省略一些代码
	}
	else
	{
		bool bDeleted = ReceivedNextBunch( Bunch, bOutSkipAck );

		if ( Bunch.IsError() )
		{
			UE_LOG( LogNetTraffic, Error, TEXT( "UChannel::ReceivedRawBunch: Bunch.IsError() after ReceivedNextBunch 1" ) );
			return;
		}

		if (bDeleted)
		{
			return;
		}

		// Dispatch any waiting bunches.
		while( InRec )
		{
			// We shouldn't hit this path on 100% reliable connections
			check( !Connection->IsInternalAck() );

			if( InRec->ChSequence!=Connection->InReliable[ChIndex]+1 )
				break;
			UE_LOG(LogNetTraffic, Log, TEXT("      Channel %d Unleashing queued bunch"), ChIndex );
			FInBunch* Release = InRec;
			InRec = InRec->Next;
			NumInRec--;
			
			// Just keep a local copy of the bSkipAck flag, since these have already been acked and it doesn't make sense on this context
			// Definitely want to warn when this happens, since it's really not possible
			bool bLocalSkipAck = false;

			bDeleted = ReceivedNextBunch( *Release, bLocalSkipAck );

			if ( bLocalSkipAck )
			{
				UE_LOG( LogNetTraffic, Warning, TEXT( "UChannel::ReceivedRawBunch: bLocalSkipAck == true for already acked packet" ) );
			}

			if ( Bunch.IsError() )
			{
				UE_LOG( LogNetTraffic, Error, TEXT( "UChannel::ReceivedRawBunch: Bunch.IsError() after ReceivedNextBunch 2" ) );
				return;
			}

			delete Release;
			if (bDeleted)
			{
				return;
			}
			//AssertInSequenced();
		}
	}
}

遍历这个InRec链表的时候，如果发现是下一个期望处理的可靠Bunch，则使用ReceivedNextBunch来处理并更新Connection->InReliable[ChIndex]。如果发现连号中断，也就是InRec->ChSequence!=Connection->InReliable[ChIndex]+1，那么就可以中止遍历了。

流量控制

由于OutRec链表的最大大小被限制为了RELIABLE_BUFFER,所以在UChannel::SendBunch( FOutBunch* Bunch, bool Merge )发送数据执行完拆包之后，会检查这个OutRec链表是否能继续发送当前的可靠包，如果会超出RELIABLE_BUFFER的限制，则会导致连接中断:

enum { RELIABLE_BUFFER = 256 }; // Power of 2 >= 1.

FPacketIdRange UChannel::SendBunch( FOutBunch* Bunch, bool Merge )
{
	// 前面的代码负责将Bunch执行可能的拆包操作，并最终放到OutgoingBunches数组里
	FPacketIdRange PacketIdRange;

	const bool bOverflowsReliable = (NumOutRec + OutgoingBunches.Num() >= RELIABLE_BUFFER + Bunch->bClose);

	if ((GCVarNetPartialBunchReliableThreshold > 0) && (OutgoingBunches.Num() >= GCVarNetPartialBunchReliableThreshold) && !Connection->IsInternalAck())
	{
		if (!bOverflowsReliable)
		{
			UE_LOG(LogNetPartialBunch, Log, TEXT("	OutgoingBunches.Num (%d) exceeds reliable threashold (%d). Making bunches reliable. Property replication will be paused on this channel until these are ACK'd."), OutgoingBunches.Num(), GCVarNetPartialBunchReliableThreshold);
			Bunch->bReliable = true;
			bPausedUntilReliableACK = true;
		}
		else
		{
			// The threshold was hit, but making these reliable would overflow the reliable buffer. This is a problem: there is just too much data.
			UE_LOG(LogNetPartialBunch, Warning, TEXT("	OutgoingBunches.Num (%d) exceeds reliable threashold (%d) but this would overflow the reliable buffer! Consider sending less stuff. Channel: %s"), OutgoingBunches.Num(), GCVarNetPartialBunchReliableThreshold, *Describe());
		}
	}

	if (Bunch->bReliable && bOverflowsReliable)
	{
		UE_LOG(LogNetPartialBunch, Warning, TEXT("SendBunch: Reliable partial bunch overflows reliable buffer! %s"), *Describe() );
		UE_LOG(LogNetPartialBunch, Warning, TEXT("   Num OutgoingBunches: %d. NumOutRec: %d"), OutgoingBunches.Num(), NumOutRec );
		PrintReliableBunchBuffer();

		// Bail out, we can't recover from this (without increasing RELIABLE_BUFFER)
		FString ErrorMsg = NSLOCTEXT("NetworkErrors", "ClientReliableBufferOverflow", "Outgoing reliable buffer overflow").ToString();
		FNetControlMessage<NMT_Failure>::Send(Connection, ErrorMsg);
		Connection->FlushNet(true);
		Connection->Close();

		return PacketIdRange;
	}
	// 后面的代码负责执行这些拆分好的Bunch的发送SendRawBunch
}

这里有一段比较特殊的逻辑，就是如果当前要发送的Bunch是非可靠Bunch且切分之后的PartialBunch数量超过了配置的GCVarNetPartialBunchReliableThreshold，就会将这些非可靠Bunch转化为可靠Bunch，当然这个的前提是变成可靠Bunch之后不会导致OutRec链表超出RELIABLE_BUFFER的限制。这里还会设置bPausedUntilReliableACK = true,这个字段的作用是暂停所有Actor属性复制的Diff，直到接收到所有发送出去的可靠包的ACK:

uint32				bPausedUntilReliableACK:1; // Unreliable property replication is paused until all reliables are ack'd.

int64 UActorChannel::ReplicateActor()
{
	SCOPE_CYCLE_COUNTER(STAT_NetReplicateActorTime);

	// 省略很多的代码

	if (bPausedUntilReliableACK)
	{
		if (NumOutRec > 0)
		{
			return 0;
		}
		bPausedUntilReliableACK = 0;
		UE_LOG(LogNet, Verbose, TEXT("ReplicateActor: bPausedUntilReliableACK is ending now that reliables have been ACK'd. %s"), *Describe());
	}
	// 省略很多的代码
}

所以UE里的流量控制是基于可靠Bunch的，非可靠Bunch只是简单的放到OutgoingBunches数组里等待发送，不会影响到可靠Bunch的流量控制。但是一旦发现可靠Bunch的未ACK数量太多，则会通过中止Actor的属性同步的方式来降低非可靠Bunch的发送频率。因为可靠Bunch是不能丢弃的，而属性同步是基于Diff的，停止一段时间属性同步之后再重新开始Diff不会影响客户端的属性正确性。

数据序列化与RPC

业务数据包的逻辑处理

在前一章中我们介绍了游戏服务器中基础的网络通信是如何实现的，在完成了网络通信之后，业务逻辑层只需要考虑业务数据包的发送和接收，而不需要再去关心底层网络的封包、解包、加密、可靠性等各种问题。但是，此时业务层对接的数据包信息含量还是太低了，只有一个std::string，至于这个数据包内所包含的内容含义，还需要业务层自己做一层数据格式封装。最简单的封装莫过于在这个数据包的前四个字节填充这个数据包的类型，此时网络层对业务逻辑开放的接口则简化为了下面的代码：

struct packet
{
	std::uint32_t cmd;
	std::string detail;
};
void send_packet(const packet& out_packet);
void handle_packet(const packet& in_packet);

业务逻辑里的每一条指令负责自己将所要传递的数据封装为packet格式，同时接收端要能够对等的通过cmd将数据投递到相应的处理函数，处理函数要知道如何解析这个数据包。在逻辑层整个数据编码、指令路由、数据解码的全套过程等价于实现一个RPC(Remote Procedure Call)。下面我们来展示一个最简单的RPC的例子：

enum rpc_cmd
{
	req_auth = 1,
	res_auth = 2,
};
// 客户端调用try_auth接口 提供账户名和密码来进行鉴权
// 函数逻辑内部负责将 (rpc_cmd::send_auth_req, (name ,passwd))这两元组进行编码 并调用网络接口进行发送
void send_auth_req(const std::string& name, const std::string& passwd)
{
	packet cur_packet;
	cur_packet.cmd = uint32_t(rpc_cmd::req_auth);
	cur_packet.detail = encode(name, passwd);
	send_packet(cur_packet);
}



// 服务器处理业务指令包的函数，对业务包 进行分发
void server_handle_msg(const std::uint32& cmd, const std::string& detail)
{
	if(cmd == int32_t(rpc_cmd::req_auth))
	{
		std::string name, passwd;
		std::string auth_res;
		bool decode_suc = decode(detail, name, passwd);
		if(!decode_suc)
		{
			auth_res = "decode fail";
		}
		else
		{
			// 数据库检查账户密码是否匹配
			auth_res = db_try_auth(name, passwd);
		}
		packet cur_packet;
		cur_packet.cmd = uint32_t(rpc_cmd::res_auth);
		cur_packet.detail = encode(auth_res);
		send_packet(cur_packet);
	}
}

// 客户端处理业务指令包的函数，对业务包 进行分发
void client_handle_msg(const std::uint32& cmd, const std::string& detail)
{
	if(cmd == int32_t(rpc_cmd::res_auth))
	{
		std::string auth_res;
		bool decode_suc = decode(detail, auth_res);
		if(!decode_suc)
		{
			auth_res = "decode fail";
		}
		if(auth_res.empty())
		{
			// 走后续的登录流程
		}
	}
}

上面的代码中完全忽略掉了网络层的收发packet实现，客户端和服务端的包分发函数都通过cmd字段来一一分发具体的数据包逻辑。业务逻辑非常清晰，但美中不足的是我们这里没有提供一个具体的encode,decode数据包内容的实现，而这部分则是数据序列化的范畴。同时这种通过handle_msg函数来集中式处理所有数据包的形式在面对海量的数据包类型时凸显了其不可维护的缺陷，每次添加一个新的RPC指令都需要写一大段类似的decode后再继续处理的逻辑。为了减轻这种维护性负担我们需要采用某种RPC注册的框架来自动对RPC数据进行路由和解包。

基于Json的RPC

封装JSON序列化接口

目前网络上流行最广的数据容器就是Javascript语言里定义的JSON(JavaScript Object Notation)，JSON内部可以包含如下五种数据：

1. null，代表无数据
2. 布尔类型，对应bool
3. 整数类型，对应std::int64_t
4. 浮点类型，对应double
5. 字符串，对应std::string
6. 数组类型，对应std::vector<json>
7. 字典类型，对应std::map<std::string, json>

有了上述类型，基本可以表示业务逻辑所需数据。这里需要注意的一点是Json标准是不支持int64这么大的数字的，它最大只支持到$2^{5}3-1$，如果超过最大值可能会导致Json的解析出错误。不过这种问题基本只会在javascript语言里出现，如果数据交换的两端都是以cpp实现的，现有的主流的json cpp库基本都可以支持到numeric_limits<std::int64_t>::max()和对应的min。这里我才用使用最广且易用性最强的nlohmann::json来作为mosaic_game的json序列化库。

指定了json序列化库之后，我们开始来实现之前引用到的encode,decode函数，由于这两个函数能够支持各种类型的参数，所以我们将这两个函数实现为变参模板函数。

template <typename T>
json encode_impl(const T& arg);
template <typename... Args>
json encode_multi(const Args&... args);

template <typename... Args>
std::string encode(const Args&... args)
{
	if constexpr(sizeof...(Args) == 0)
	{
		return "null";
	}
	if constexpr(sizeof...(Args) == 1)
	{
		return encode_impl<Args>(args).to_string();
	}
	return encode_multi<Args...>(args).to_string();
}

template <typename T>
bool decode_impl(const json& data, T& dest);
template <typename... Args>
bool decode_multi(const json& data, Args&... args);

template<typename... Args>
bool decode(const std::string& data, Args&... args)
{
	if(!json::accept(data))
	{
		return false;
	}
	auto temp_j = json::parse(data);
	if constexpr(sizeof...(Args) == 0)
	{
		return true;
	}
	if constexpr(sizeof...(Args) == 1)
	{
		return decode_impl<Args>(temp_j, args);
	}
	return decode_multi<Args...>(temp_j, args);
}

由于decode_impl基本等价于encode_impl的逆操作，所以这里只详细介绍encode_impl的实现。对于json规范中规定的物种基本类型，encode_impl分别进行了特化，直接返回原始数据对应的json。对于STL里规定的相关类型，按照类型的具体语义进行特化：

// forward declare
template <typename T>
json encode_impl(const std::optional<T>& data); // data有效则返回encode_impl<T> 否则返回bull
template <typename T>
json encode_impl(const std::vector<T>& data); // 构造一个json vector 然后遍历每个元素调用encode_impl<T>进行push_back
template <typename T1, typename T2>
json encode_impl(const std::pair<T1, T2>& data); //构造两个元素的json vector 分别调用encode_impl<T1> encode_impl<T2>来填充0， 1 两个元素
template <typename T1, std::size_t T2>
json encode_impl(const std::array<T1, T2>& data); // 等价于调用encode_impl<std::vector<T>>()
template <typename... args>
json encode_impl(const std::tuple<args...>& data); // 构造一个json vector 然后对data里每个元素调用encode_impl<args>然后执行push_back
template <typename... Args>
json encode_impl(const std::variant<Args...>& data); // 如果没有有效数据则返回null 否则返回对应具体类型的encode_impl<T>
template <typename T1, typename T2>
json encode_impl(const std::map<T1, T2>& data); // 如果T1是string 则构造一个json object进行序列化，否则当作std::vector<std::pair<T1, T2>>来encode_impl
template <typename T1, typename T2>
json encode_impl(const std::unordered_map<T1, T2>& data); // 当作std::vector<std::pair<T1, T2>>来encode_impl
template <typename T>
json encode_impl(const std::unordered_map<std::string, T>& data); // 构造一个json object进行序列化
template <typename T1, typename T2>
json encode_impl(const std::multimap<T1, T2>& data); // 当作std::vector<std::pair<T1, T2>>来encode_impl
template <typename T1, typename T2>
json encode_impl(const std::unordered_multimap<T1, T2>& data); // 当作std::vector<std::pair<T1, T2>>来encode_impl
template <typename T1>
json encode_impl(const std::set<T1>& data); // 当作std::vector<T1>来encode_impl
template <typename T1>
json encode_impl(const std::unordered_set<T1>& data);// 当作std::vector<T1>来encode_impl
template <typename T1>
json encode_impl(const std::multiset<T1>& data);// 当作std::vector<T1>来encode_impl
template <typename T1>
json encode_impl(const std::unordered_multiset<T1>& data);// 当作std::vector<T1>来encode_impl
template <typename... Args>
json encode_multi(const Args&... args); // 构造一个json vector 遍历args里所有元素进行encode_impl 后push_back

完成了encode的逻辑之后，将对应类型的encode_impl操作进行逆操作即可实现decode_impl，这里就不再赘述。

Json RPC的注册

开头的样例RPC代码中，所有的RPC处理都需要在server_handle_msg, client_handle_msg中加入若干行反序列化代码再转接到真正处理对应RPC的业务函数。随着RPC数量的增多，多人同时编辑此函数导致了维护性急剧下降，为解决这个问题，我们将转向注册制的RPC处理:

class rpc_handler
{
	std::unordered_map<std::string, std::function<void(const json&)>> m_registered_rpcs;
	void handle_rpc(const std::string& cmd, const std::string& detail)
	{
		
		auto temp_iter = g_registered_rpcs.find(cmd);
		if(temp_iter == g_registered_rpcs.end())
		{
			return;
		}
		if(!json::accept(detail))
		{
			return;
		}
		temp_iter->second()(json::parse(detail));
	}
	void register_rpc(const std::string& cmd, std::function<void(const json&)> cmd_handler)
	{
		assert(g_registered_rpcs.find(cmd) == g_registered_rpcs.end());
		m_registered_rpcs[cmd] = cmd_handler;
	}
};

我们采取类似于上面的rpc_handler类型来处理rpc的注册与分发，与此同时修改rpc_cmd的类型从enum改为std::string。这样新添加任意一个RPC都不需要修改公共的文件，只需要在对应RPC逻辑的cpp文件中调用对应的register_rpc函数即可。这种设计极大的减轻了RPC框架维护的心智负担。

void handle_auth_req(const std::string& name, const std::string& passwd);
g_rpc_handler.register_rpc("auth_req", [](const json& data)
{
	std::string name, passwd;
	if(!decode(data, name, passwd))
	{
		return;
	}
	handle_auth_req(name, passwd);
});

基于Schema的RPC

在前面一节内容中我们基于Json序列化和注册机制实现了一个非常简单易用的RPC框架，这个框架极大的减轻了RPC处理代码编写逻辑复杂度，但是每个RPC都需要手动编写胶水代码进行注册这种形式仍然有很大的改进空间。如果想避免掉手写注册代码这种繁杂劳动，我们可以求助于一些基于Schema的RPC框架，即通过某种接口描述语言(IDL-Interface Description Language)来描述所有的RPC接口规范，然后使用框架附带的工具自动生成多种语言的序列化、反序列化、接口注册、接口分发等相关代码，使得使用者可以完全不考虑框架内部细节，只需要关心业务逻辑。目前主流的基于Schema的RPC框架主要有Google出品的grpc和FaceBook出品的Thrift，下面我们分别来介绍。

grpc

grpc的IDL使用的是Google开源的protobuf格式。protobuf是一个跨语言、跨平台的序列化协议，下面是一个非常简单的基于protobuf的消息格式声明：

message SearchRequest {
	string query = 1;
	int32 page_number = 2;  // Which page number do we want?
	int32 results_per_page = 3;  // Number of results to return per page.
}

上面的结构体中，每个字段的左边都有其类型说明符，右边则带有一个从1开始计数的不重复编号。protobuf对于基础数据类型的支持与Json类似，但是对于int64,uint64的支持则是全范围的。对于复杂类型的支持，protobuf比Json更完善一些，提供了如下四种:

1. optional 可空描述符，代表该字段在具体的消息中不一定存在对应的数据
2. repeated 数组描述符，代表该字段其实是一个数组，等价于Json::Vector
3. map<K,V> 字典描述符，代表该字段是一个字典，这里对于K的类型并没有像Json一样限定为string
4. enum 枚举描述符，对应cpp的一个枚举类

下面的代码片段展示了protobuf对于这些容器类型数据结构的支持样例:

message Person {
	optional string name = 1;
	optional int32 id = 2;
	optional string email = 3;

	enum PhoneType {
		PHONE_TYPE_UNSPECIFIED = 0;
		PHONE_TYPE_MOBILE = 1;
		PHONE_TYPE_HOME = 2;
		PHONE_TYPE_WORK = 3;
	}

	message PhoneNumber {
		optional string number = 1;
		optional PhoneType type = 2 [default = PHONE_TYPE_HOME];
	}

	repeated PhoneNumber phones = 4;
}

message AddressBook {
	repeated Person people = 1;
}

有了消息结构的声明之后，我们可以使用protobuf自带的预处理工具protoc来生成各种语言版本的胶水代码。这里我们使用一个包含上面的Person的文件作为输入：

protoc -I=$SRC_DIR --cpp_out=$DST_DIR $SRC_DIR/xxx.proto
protoc --cpp_out=. person.proto 

在这个指令执行之后，会在当前目录下生成person.pb.h与person.pb.cc两个文件，h文件作为接口文件,cc文件作为实现文件。上面短短的消息声明触发生成的两个文件包含了近2000行不会有人想去读的代码，以此为代价换来的是接口的全面。不过对于用户而言，在Person类上只关心这么几个接口就行：

	// name
	inline bool has_name() const;
	inline void clear_name();
	inline const ::std::string& name() const;
	inline void set_name(const ::std::string& value);
	inline void set_name(const char* value);
	inline ::std::string* mutable_name();

	// id
	inline bool has_id() const;
	inline void clear_id();
	inline int32_t id() const;
	inline void set_id(int32_t value);

	// email
	inline bool has_email() const;
	inline void clear_email();
	inline const ::std::string& email() const;
	inline void set_email(const ::std::string& value);
	inline void set_email(const char* value);
	inline ::std::string* mutable_email();

	// phones
	inline int phones_size() const;
	inline void clear_phones();
	inline const ::google::protobuf::RepeatedPtrField< ::tutorial::Person_PhoneNumber >& phones() const;
	inline ::google::protobuf::RepeatedPtrField< ::tutorial::Person_PhoneNumber >* mutable_phones();
	inline const ::tutorial::Person_PhoneNumber& phones(int index) const;
	inline ::tutorial::Person_PhoneNumber* mutable_phones(int index);
	inline ::tutorial::Person_PhoneNumber* add_phones();

上面的代码里包含了一般用户使用的所有相关字段的get,set接口，此外在这个类型继承的基类上则提供了序列化与反序列化的接口：

bool SerializeToString(string* output) const;
bool ParseFromString(const string& data);

其序列化接口生成的字符串是一个二进制字符串，不像Json生成的是一个人机皆可读的文本字符串。采用二进制进行编码的好处就是使序列化之后的数据长度更少，这样在网络上传递这些消息时所需的带宽就会降低很多。

protobuf不仅仅能声明消息结构，还支持声明RPC服务接口，下面是一个最简单的使用其IDL声明RPC接口的样例：

// The greeter service definition.
service Greeter {
	// Sends a greeting
	rpc SayHello (HelloRequest) returns (HelloReply) {}
}

// The request message containing the user's name.
message HelloRequest {
	string name = 1;
}

// The response message containing the greetings
message HelloReply {
	string message = 1;
}

这里我们需要走两遍protoc分别生成消息协议代码和rpc service代码：

$ protoc --cpp_out=. ./greeter.proto
$ protoc  --grpc_out=. --plugin=protoc-gen-grpc ./greeter.proto

在上面的第二行指令会导致生成greeter.grpc.pb.h, greeter.grpc.pb.cc两个额外的文件，这两个文件里包含了300行依然无可读性的生成代码。这些代码里对于用户接口来说重要的只有这么几行：

class Greeter final {
	class Service : public ::grpc::Service {
	 public:
		Service();
		virtual ~Service();
		// Sends a greeting
		virtual ::grpc::Status SayHello(::grpc::ServerContext* context, const ::HelloRequest* request, ::HelloReply* response);
	};
class Stub final : public StubInterface {
	 public:
		Stub(const std::shared_ptr< ::grpc::ChannelInterface>& channel, const ::grpc::StubOptions& options = ::grpc::StubOptions());
		::grpc::Status SayHello(::grpc::ClientContext* context, const ::HelloRequest& request, ::HelloReply* response) override;
	};
};

Greeter::Service提供一些虚接口来承接服务器接收到相关RPC数据之后的业务层逻辑，具体的实现需要业务方继承这个类型实现所有的虚接口：

class GreeterServiceImpl final : public Greeter::Service {
	Status SayHello(ServerContext* context, const HelloRequest* request,
									HelloReply* reply) override {
		std::string prefix("Hello ");
		reply->set_message(prefix + request->name());
		return Status::OK;
	}
};

实现了业务层逻辑之后，我们可以把这个Service注册到一个开启好了的grpc server上去，grpc内部负责处理网络收发、数据解析、RPC分发等框架层逻辑：

void RunServer(uint16_t port) {
	std::string server_address = absl::StrFormat("0.0.0.0:%d", port);
	GreeterServiceImpl service;

	grpc::EnableDefaultHealthCheckService(true);
	grpc::reflection::InitProtoReflectionServerBuilderPlugin();
	ServerBuilder builder;
	// Listen on the given address without any authentication mechanism.
	builder.AddListeningPort(server_address, grpc::InsecureServerCredentials());
	// Register "service" as the instance through which we'll communicate with
	// clients. In this case it corresponds to an *synchronous* service.
	builder.RegisterService(&service);
	// Finally assemble the server.
	std::unique_ptr<Server> server(builder.BuildAndStart());
	std::cout << "Server listening on " << server_address << std::endl;

	// Wait for the server to shutdown. Note that some other thread must be
	// responsible for shutting down the server for this call to ever return.
	server->Wait();
}

int main(int argc, char** argv) {
	absl::ParseCommandLine(argc, argv);
	RunServer(absl::GetFlag(FLAGS_port));
	return 0;
}

Greeter::Stub则负责提供给客户端作为相关RPC调用的入口:

class GreeterClient {
 public:
	GreeterClient(std::shared_ptr<Channel> channel)
			: stub_(Greeter::NewStub(channel)) {}


	std::string SayHello(const std::string& user) {
		// Follows the same pattern as SayHello.
		HelloRequest request;
		request.set_name(user);
		HelloReply reply;
		ClientContext context;

		Status status = stub_->SayHello(&context, request, &reply);
		if (status.ok()) {
			return reply.message();
		} else {
			std::cout << status.error_code() << ": " << status.error_message()
								<< std::endl;
			return "RPC failed";
		}
	}
	 private:
	std::unique_ptr<Greeter::Stub> stub_;
};

有了这个封装好的GreeterClient之后，往指定grpc server发送一次RPC请求就很简单了：

int main(int argc, char** argv) {
	absl::ParseCommandLine(argc, argv);
	// Instantiate the client. It requires a channel, out of which the actual RPCs
	// are created. This channel models a connection to an endpoint specified by
	// the argument "--target=" which is the only expected argument.
	std::string target_str = absl::GetFlag(FLAGS_target);
	// We indicate that the channel isn't authenticated (use of
	// InsecureChannelCredentials()).
	GreeterClient greeter(
			grpc::CreateChannel(target_str, grpc::InsecureChannelCredentials()));
	std::string user("world");
	std::string reply = greeter.SayHello(user);
	std::cout << "Greeter received: " << reply << std::endl;

	return 0;
}

例如整体来说,grpc作为一个RPC框架，给用户层屏蔽了诸多细节的同时，在使用上也非常方便。通信的两端只要共享同一个proto文件，即可轻松跨平台、跨语言。但是在游戏业务内我们却极其不推荐使用grpc作为RPC解决方案，主要有如下四点:

1. grpc的框架依赖过重，前述的最简rpc样例 greeter_client greeter_server编译出来之后文件大小可达55M:

-rwxrwxr-x  1 qian qian  51M  4月 25 00:55 greeter_client
-rwxrwxr-x  1 qian qian  53M  4月 25 00:55 greeter_server

而且这两个二进制还带来了很多动态库的依赖：

qian@qian-desktop:~/Github/grpc/examples/cpp/helloworld/build$ ldd ./greeter_client
linux-vdso.so.1 (0x00007ffd58b9d000)
libsystemd.so.0 => /lib/x86_64-linux-gnu/libsystemd.so.0 (0x00007f62d9bdf000)
libstdc++.so.6 => /lib/x86_64-linux-gnu/libstdc++.so.6 (0x00007f62d8000000)
libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007f62d8315000)
libgcc_s.so.1 => /lib/x86_64-linux-gnu/libgcc_s.so.1 (0x00007f62d9bbb000)
libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f62d7c00000)
/lib64/ld-linux-x86-64.so.2 (0x00007f62d9cce000)
libcap.so.2 => /lib/x86_64-linux-gnu/libcap.so.2 (0x00007f62d9bad000)
libgcrypt.so.20 => /lib/x86_64-linux-gnu/libgcrypt.so.20 (0x00007f62d7eb8000)
liblzma.so.5 => /lib/x86_64-linux-gnu/liblzma.so.5 (0x00007f62d9b7b000)
libzstd.so.1 => /lib/x86_64-linux-gnu/libzstd.so.1 (0x00007f62d7b49000)
liblz4.so.1 => /lib/x86_64-linux-gnu/liblz4.so.1 (0x00007f62d9b58000)
libgpg-error.so.0 => /lib/x86_64-linux-gnu/libgpg-error.so.0 (0x00007f62d82ef000)

当proto文件随着RPC数量的增多，protoc生成的代码也越来越大，再加上引入的框架代码，编译时间会增加非常多。如果proto文件频繁更新，则编译时间拖慢工作流的影响会更加显著。

2. protobuf的版本冲突问题，如果一个程序连接了多个静态连接了protobuf的动态库，则使用者需要去保证这些动态库使用的protobuf的版本要强一致，否则会造成运行时crash。一旦某个动态库是由第三方团队维护的，则保证这种一致性基本是不可能的任务。在这种设计下，要确保全链路都使用grpc就只能参考Google的Mono Repo，同时禁止出现第三方的动态库里出现protobuf。

3. grpc对于网络层封装的比较重度，而游戏业务经常需要自己去管理网络层的接入与收发，例如我们在前面一章中提到的KCP，这样就产生了很多冲突。

4. protobuf的编码很多时候无法满足游戏业务尽可能的降低网络带宽的需求，因为业务逻辑经常对移动同步等高频RPC自己实现相关数据的以bit为基础的序列化协议，此时在proto文件中只能定义一个string来代表传输的数据，丧失了序列化的意义还容易触发二次编码。此外protobuf作为纯序列化库，对比flatbuffer、thrift和msgpack有不小的性能劣势。

所以，游戏业务中一般只会有限的采用protobuf作为数据序列化协议的一种，而不会整体采用grpc作为网络通信框架。

Mosaic Game 的 RPC 实现

考虑到便利性的需求，mosaic_game使用基于json的RPC方案。为了避免前述的原始Json RPC方案里的各种需要手工维护的注册和序列化反序列化代码的重复劳动，mosaic_game也使用类似于UHT的工具来自动生成这些代码。mosaic_game使用的代码生成工具与UHT的原理有很大的不同。UHT是通过纯文本扫描项目中所有头文件里带UFunction, UProperty等标记的行来获取相关反射信息，而mosaic_game使用的是libclang对相关头文件进行完整的cpp语义解析，并以Json格式导出所有带标注的类型声明信息。有了这些类型声明信息之后，我们再使用基于mustache的模板引擎，生成所需的胶水代码来进行RPC的注册和序列化。这个基于libclang的cpp代码生成工具在mosaic_game中有很多应用场景，后面有单独的一章来描述此工具的相关细节。

Mosaic Game 中的RPC声明与注册

在mosaic_game中，我们支持了在Actor/ActorComponent, Space/SpaceComponent, Manager/ManagerComponent等类型上声明RPC，这三种RPC宿主实现基本一样，所以这里以Actor/ActorComponent来作为样例来说明。下面就是一个在PlayerChatComponent上的RPC声明:

Meta(rpc) void chat_add_msg_request(const utility::rpc_msg& msg, std::uint8_t chat_type, const std::string& to_player_id, const json::object_t& detail);

上面的函数声明中，Meta(rpc)的作用是提示libclang当前函数声明是一个RPC函数。在mosaic_game中我们规定所有的RPC的声明都要遵循如下结构：

1. 左边必须以Meta(rpc)开头，以通知libclang当前函数声明是一个RPC函数
2. 函数的返回值必须为void，即所有的RPC函数都是异步调用
3. 函数的第一个参数必须为const utility::rpc_msg&类型，这个类型用来容纳当前RPC的所有信息

struct rpc_msg
{
	std::string cmd; // 当前rpc的名字
	std::string err; // 如果是响应rpc 则此处填充响应错误信息 如果无错误则保持为空
	std::vector<json> args; // 当前rpc的所有参数序列化为json之后的数组
	std::string from; // 当前rpc的调用者信息 可选
	std::uint32_t flag = 0; // 当前rpc的flag信息 例如是否允许客户端调用 是否允许服务器调用
	
	// 下面两个接口提供到json的序列化与反序列化
	friend void to_json(json& result, const utility::rpc_msg& msg);

	bool from_json(const json& data);
}

4. 函数的实际使用参数依次排在第一个参数之后

有了这个RPC函数声明之后，我们就可以使用libclang进行类型信息导出与胶水代码生成，针对每个带有RPC函数声明的xxx.h头文件，我们都会生成一个xxx.rpc.incpp文件，插入到xxx.cpp之中：

// player_chat_component.cpp
#include "component/player/player_chat_component.h"
#include "entity/player_entity.h"
#include "player_chat_component.rpc.incpp"

而这个xxx.rpc.incpp文件主要提供了一个内部类型xxx::rpc_helper的完整定义：

// player_chat_component.rpc.incpp
namespace spiritsaway::mosaic_game::entity
{
	class player_chat_component::rpc_helper
	{
		static utility::rpc_msg::call_result chat_add_msg_request(player_chat_component* rpc_owner, const spiritsaway::utility::rpc_msg& cur_rpc)
		{
			std::remove_cv_t<std::remove_reference_t<unsigned char>> chat_type;
			std::remove_cv_t<std::remove_reference_t<const std::string&>> to_player_id;
			std::remove_cv_t<std::remove_reference_t<const json::object_t&>> detail;
			if(!spiritsaway::serialize::decode_multi(cur_rpc.args,  chat_type , to_player_id , detail ))
			{
				return utility::rpc_msg::call_result::invalid_format;
			}
			rpc_owner->chat_add_msg_request(cur_rpc,  chat_type , to_player_id , detail );
			

			return utility::rpc_msg::call_result::suc;
		}
		constexpr static std::uint32_t flag_for_chat_add_msg_request = 0;
				static const std::unordered_map<std::string, spiritsaway::utility::rpc_cmd_info>& get_rpc_map()
		{
			static std::unordered_map<std::string, spiritsaway::utility::rpc_cmd_info> cur_rpc_map = {
				{ "chat_add_msg_notify", spiritsaway::utility::rpc_cmd_info{ 0, "(const spiritsaway::utility::rpc_msg& msg, const std::string& chat_key, unsigned int msg_seq, const json::object_t& detail)", flag_for_chat_add_msg_notify }  },
				{ "chat_add_msg_reply", spiritsaway::utility::rpc_cmd_info{ 1, "(const spiritsaway::utility::rpc_msg& msg, const std::string& chat_key, unsigned int msg_seq, const json::object_t& detail)", flag_for_chat_add_msg_reply }  },
				{ "chat_add_msg_request", spiritsaway::utility::rpc_cmd_info{ 2, "(const spiritsaway::utility::rpc_msg& msg, unsigned char chat_type, const std::string& to_player_id, const json::object_t& detail)", flag_for_chat_add_msg_request }  },
			};
			return cur_rpc_map;
		}
		static utility::rpc_msg::call_result rpc_call(player_chat_component* rpc_owner, std::uint32_t cur_rpc_idx, const spiritsaway::utility::rpc_msg& cur_rpc)
		{
			switch(cur_rpc_idx)
			{
				case 0:
					return chat_add_msg_notify(rpc_owner, cur_rpc);
				case 1:
					return chat_add_msg_reply(rpc_owner, cur_rpc);
				case 2:
					return chat_add_msg_request(rpc_owner, cur_rpc);
				default:
					return utility::rpc_msg::call_result::rpc_not_found;
			}
		}
		static utility::rpc_msg::call_result rpc_call(player_chat_component* rpc_owner, const spiritsaway::utility::rpc_msg& cur_rpc)
		{
			auto& cur_rpc_map = get_rpc_map();
			auto cur_iter = cur_rpc_map.find(cur_rpc.cmd);
			if(cur_iter == cur_rpc_map.end())
			{
				return utility::rpc_msg::call_result::rpc_not_found;
			}
			if(cur_rpc.flag && (cur_rpc.flag &cur_iter->second.cmd_flag) == 0)
			{
				return utility::rpc_msg::call_result::flag_not_meet;
			}
			return rpc_call(rpc_owner, std::uint32_t(cur_iter->second.cmd_idx), cur_rpc);
		}
	};
}

除了这个辅助类型的完整定义，这个rpc.incpp文件里还提供了如下两个函数的定义：

const std::unordered_map<std::string, spiritsaway::utility::rpc_cmd_info>& player_chat_component::get_rpc_indexes() const
{
	return rpc_helper::get_rpc_map();
}

utility::rpc_msg::call_result player_chat_component::rpc_component_on_rpc(std::uint32_t cur_rpc_idx, const utility::rpc_msg& cur_msg)
{
	return rpc_helper::rpc_call(this, cur_rpc_idx, cur_msg);
}

这两个函数的声明提供在base_component上，调用方则在component_owner上：

virtual utility::rpc_msg::call_result component_owner::rpc_owner_on_rpc(const utility::rpc_msg& cur_msg)
{
	// 通过rpc名字查找该rpc的注册信息
	auto cur_iter = m_component_register_info->component_rpcs.find(cur_msg.cmd);
	if(cur_iter == m_component_register_info->component_rpcs.end())
	{
		return utility::rpc_msg::call_result::rpc_not_found;
	}
	// 找到之后判断flag是否满足 例如是否允许客户端调用
	if(cur_msg.flag && (cur_msg.flag & cur_iter->second.cmd_flag) == 0)
	{
		return utility::rpc_msg::call_result::flag_not_meet;
	}
	// 这里的cmd_idx是一个uint64_t 由两个32位整数拼接而成
	// 高32位代表所属component在component_owner的components数组的索引
	// 低32位则是这个rpc在对应component里的索引
	std::uint64_t rpc_idx = cur_iter->second.cmd_idx & 0xffffffff;
	auto cur_component_idx = (cur_iter->second.cmd_idx >> 32) & 0xffffffff;
	auto cur_comp = get_component(std::uint32_t(cur_component_idx));
	if(!cur_comp)
	{
		return utility::rpc_msg::call_result::rpc_not_found;
	}
	// 找到对应component之后 调用base_component上提供的虚方法rpc_component_on_rpc 
	// 内部实现就是通过switch case来执行对应的rpc函数
	return cur_comp->rpc_component_on_rpc(std::uint32_t(rpc_idx), cur_msg);
}

上面的RPC信息查找依赖于component_owner上维护好的m_component_register_info->component_rpcs信息，每个base_component在绑定到一个component_owner上之后，会将此base_component上的所有rpc都注册到component_rpcs这个 map中：

// 当一个component添加到某个owner之后会调用此函数
virtual void base_component::on_set_owner()
{
	m_owner->add_component_rpcs(get_rpc_indexes(), m_component_type_id);
}
void component_owner::add_component_rpcs(const std::unordered_map<std::string, rpc_cmd_info>& rpc_names, const std::uint32_t component_type_id)
{
	auto& component_rpc_registered = m_component_register_info->component_rpc_registered;
	if(component_type_id >= component_rpc_registered.size())
	{
		component_rpc_registered.resize(component_type_id + 1);
	}
	// 避免重复注册
	if(!component_rpc_registered[component_type_id])
	{
		for(const auto& one_rpc_name: rpc_names)
		{
			// 这里拼接两个uint32为一个uint64
			m_component_register_info->component_rpcs[one_rpc_name.first] = rpc_cmd_info{(std::uint64_t(component_type_id)<<32) + one_rpc_name.second.cmd_idx, one_rpc_name.second.cmd_info};
		}
	}
	
}

通过上述base_component component_owner rpc_helper的配合，整体的一个RPC注册与调用流程就基本跑通了。

Mosaic Game 中的RPC序列化

上面一节的内容描述了一个Actor及相关ActorComponent上的所有RPC函数的注册与调用，为了让网络另外一端的特定Actor对象执行指定RPC，我们还需要解决RPC的序列化问题。此时我们来回顾一下之前我们构造的一个RPC封装结构：

struct rpc_msg
{
	std::string cmd; // 当前rpc的名字
	std::string err; // 如果是响应rpc 则此处填充响应错误信息 如果无错误则保持为空
	std::vector<json> args; // 当前rpc的所有参数序列化为json之后的数组
	std::string from; // 当前rpc的调用者信息 可选
	std::uint32_t flag = 0; // 当前rpc的flag信息 例如是否允许客户端调用 是否允许服务器调用
	
	// 下面两个接口提供到json的序列化与反序列化
	friend void to_json(json& result, const utility::rpc_msg& msg);

	bool from_json(const json& data);
}

从这个结构体的成员变量声明可以看出，我们需要将RPC的所有参数都转换为Json，然后依次填充到args字段上，同时将RPC的名字填充到cmd字段上。这里为了方便填充所有的RPC参数，提供了一个基于模板的set_args接口，这个接口会将传入的所有变参都调用开头提到的json encode接口自动转换为json类型，其实就等价于encode_multi：

template <typename... Args>
void set_args(Args&&... in_args)
{
	args.reserve(sizeof...(Args));
	(args.push_back(encode<Args>(std::forward<Args>(in_args))),...);
}

通过encode这个模板函数，我们不仅支持了那些本来就支持自动转换为json的基本类型，还支持了带json encode() const接口的任意类型作为RPC参数。

所有字段都填充好之后，再将这个结构体序列化为Json，这个序列化流程比较简单。Json可以进一步序列化为字符串，转变为字符串之后就可以直接进行网络数据传递，投送到网络另外一端的指定Actor，对应的Actor会将此字符串反序列化为Json，并调用rpc_msg::from_json来进行字段初始化，成功初始化之后，通过rpc_owner_on_rpc(const utility::rpc_msg& cur_msg)进行分发，调用xxx::rpc_heler上通过libclang生成的辅助函数，这个辅助函数则负责解析出当前RPC的所有参数，并调用最终执行的函数。这里的参数解析使用了前面提到的decode_multi，等价于encode_multi的逆过程。

class player_chat_component::rpc_helper
{
	static utility::rpc_msg::call_result chat_add_msg_request(player_chat_component* rpc_owner, const spiritsaway::utility::rpc_msg& cur_rpc)
	{
		std::remove_cv_t<std::remove_reference_t<unsigned char>> chat_type;
		std::remove_cv_t<std::remove_reference_t<const std::string&>> to_player_id;
		std::remove_cv_t<std::remove_reference_t<const json::object_t&>> detail;
		if(!spiritsaway::serialize::decode_multi(cur_rpc.args,  chat_type , to_player_id , detail ))
		{
			return utility::rpc_msg::call_result::invalid_format;
		}
		rpc_owner->chat_add_msg_request(cur_rpc,  chat_type , to_player_id , detail );
		

		return utility::rpc_msg::call_result::suc;
	}
};

实际使用中发现RPC的序列化数据中，cmd字段占据的长度相当可观。因为我们在给RPC赋予名字时，为了表达足够的信息，会不可避免的将名字弄得很长。为了降低RPC名字所需要的网络流量，我们参考Unreal Engine里RPC的实现， 将RPC的名字转换为一个uint32索引。为了实现这个名字索引机制，我们在mosaic_game中维护了一个常用术语表vector<string> cmd_vec，基于这个术语表构造出名字字符串与名字索引之间的映射std::unordered_map<std::string, std::uint32_t> cmd_map;：

void rpc_msg::init_cmd_vec(const std::vector<std::string>& in_cmd_vec)
{
	assert(cmd_vec.empty());
	cmd_vec.push_back({});
	cmd_vec.insert(cmd_vec.end(), in_cmd_vec.begin(), in_cmd_vec.end());
	cmd_vec = in_cmd_vec;
	for(std::uint32_t i = 0; i< in_cmd_vec.size(); i++)
	{
		cmd_map[in_cmd_vec[i]] = i;
	}
}
std::string rpc_msg::seq_to_cmd(std::uint32_t seq)
{
	if(seq >= cmd_vec.size())
	{
		return {};
	}
	return cmd_vec[seq];
}

std::uint32_t rpc_msg::cmd_to_seq(const std::string& cmd)
{
	auto temp_iter = cmd_map.find(cmd);
	if(temp_iter == cmd_map.end())
	{
		return 0;
	}
	return temp_iter->second;
}

有了这个cmd_map cmd_vec之后，rpc_msg的序列化与反序列化需要做相应的修改：

void to_json(json& result, const utility::rpc_msg& msg)
{
	auto temp_seq = utility::rpc_msg::cmd_to_seq(msg.cmd);
	if(temp_seq == 0)
	{
		result["cmd"] = msg.cmd;
	}
	else
	{
		result["cmd"] = temp_seq;
	}
	// 其他字段的序列化
}
bool rpc_msg::from_json(const json& data)
{

	auto temp_cmd_iter = data.find("cmd");
	if(temp_cmd_iter == data.end())
	{
		return false;
	}
	if(temp_cmd_iter->is_string())
	{
		temp_cmd_iter->get_to(cmd);
	}
	else if(temp_cmd_iter->is_number_unsigned())
	{
		auto temp_seq = temp_cmd_iter->get<std::uint32_t>();
		if(temp_seq)
		{
			cmd = seq_to_cmd(temp_seq);
		}
		else
		{
			return false;
		}
	}
	else
	{
		return false;
	}
	// 其他字段的反序列化
}

整个常用名字术语表的维护则通过我们前述的libclang工具来完成的，大大的避免了人工处理时容易出现的各种纰漏。

Mosaic Game 远程消息投递

游戏服务器一般来说是由承担不同功能角色一组进程的集合，这些进程可以放在同一台物理机上，也可以分布式的存在于多台通过网络互联的物理机中。游戏服务器内管理着各种可以通信的对象，分布在这些进程之中。同时这些可通信的对象与进程之间的关系是动态的，可能会按照业务需求将某个对象从物理机器A中的某个进程切换到物理机器B中的某个进程。所以在游戏服务器中的对象间通信不能直接使用ip:port::target_id这种静态的配置形式，需要在业务层封装好一个基于对象名字的虚拟通道channel消息投递机制，以应对对象的进程间迁移。同时由于游戏内的玩家可能随时上线下线，所以对于玩家的某些消息还需要额外考虑其不在线的情况 ，使用持久化的数据库来确保玩家对这些持久化消息的有序接收。接下来我们将介绍一下mosaic_game中提供的面向不同应用场景的各种消息投递机制，以及相关应用组件。

在线消息投递

在线消息投递处理的是向目前在线的服务端entity,service发送rpc消息的过程。如果发送时目标不在线，则消息投递失败；如果在投递过程中目标下线，则消息投递失败。由于此类消息不保证能够可靠的通知到目标，所以此类消息一般都是用来执行客户端消息通知，不能依赖此类消息去修改entity的持久化数据。根据这个在线消息投递实现机制，又可以细分为三种子类型：

1. 单播消息 只向一个目标进行消息投递
2. 多播消息 向多个在线目标进行消息投递
3. 广播消息 向所有在线目标进行消息投递

单播消息

在线单播消息投递是所有消息投递实现的基础，在mosaic_game中暴露出了如下接口来支持向指定的一个对象发送在线消息：

void json_server::call(std::shared_ptr<const std::string> dest, std::shared_ptr<const std::string> msg, enums::packet_cmd cur_packet_cmd, std::uint8_t cur_packet_detail_cmd)
{
	m_router->push_msg(m_local_name_ptr, dest, msg, enums::packet_cmd_helper::encode(cur_packet_cmd, cur_packet_detail_cmd));
}

dest字段代表投递目标的唯一标识符，为server_id::target_id形式的字符串，代表server_id服务器上的target_id对应的发送目标，后面的::target_id某些情况下可以省略。然后msg字段就是要发送的消息的字节流，这里使用shared_ptr<const std::string>是为了避免出现字符串的拷贝操作，同时也能更好的支持多播。

参数里剩下的cur_packet_cmd与cur_packet_detail_cmd可以理解为消息类型和在这个消息下的消息子类型，编码时会将这两个字段合并为一个uint16进行处理：

enum class packet_cmd: std::uint8_t
{
	server_control = 0,
	client_to_game,
	game_to_client,
	server_rpc_msg,
	server_raw_msg,
	entity_msg,
	actor_migrate_msg,
	max,
};
struct packet_cmd_helper
{
	static std::uint16_t encode(packet_cmd in_packet_cmd, std::uint8_t in_cmd_detail)
	{
		std::uint16_t result = std::uint16_t(in_packet_cmd);
		result <<= 8;
		result += in_cmd_detail;
		return result;
	}

	static std::pair<packet_cmd, std::uint8_t> decode(std::uint16_t in_combine_cmd)
	{
		std::pair<packet_cmd, std::uint8_t> result;
		result.second = in_combine_cmd % 256;
		result.first = packet_cmd(in_combine_cmd / 256);
		return result;
	}
};

这个call接口只是对network_router::push_msg的一个简单转发，额外加上了当前发送者的进程标识符m_local_name_ptr,内部会根据这个dest找到合适的远程连接connection，并将消息添加到这个connection的发送队列中：

bool network_router::push_msg(std::shared_ptr<const std::string> from, std::shared_ptr<const std::string> dest, std::shared_ptr<const std::string> data, std::uint16_t cmd)
{

	auto cur_proxy_resource = m_anchor_collection.find_proxy_for_anchor(*dest);
	if(!cur_proxy_resource)
	{
		m_logger->error("push_msg cant find anchor_resources from {} dest {}  data {}", *from, *dest, *data);

		return false;
	}
	auto cur_proxy_con = cur_proxy_resource->get_connection();
	if (cur_proxy_con)
	{
		if (push_msg(cur_proxy_con, from, dest, data, cmd))
		{
			return true;
		}
	}
	return cur_proxy_resource->try_push(from, dest, data, cmd);
}

从dest找到对应的connection的逻辑由anchor_collection类型负责，这个类型记录了anchor锚点到网络投递资源的映射，对外暴露了名字绑定接口来添加两者之间的映射：

bool network_router::link_anchor_to_connection(const std::string& anchor_name, const net_connection* connection)
{
	auto cur_connection_iter = m_connection_resources.find(connection);
	if (cur_connection_iter == m_connection_resources.end())
	{
		return false;
	}
	if(m_anchor_collection.create_resource(anchor_name, connection, cur_connection_iter->second->output_channel))
	{
		cur_connection_iter->second->anchors.insert(anchor_name);
		return true;
	}
	else
	{
		return false;
	}
}

查询的时候查找以::分割的最长前缀去匹配。举个例子来说，如果server1::target_id找不到记录的话，则继续以server_1去查找记录，这样就可以让server_1这个connection_resource去代理所有的server_1::xxx形式的rpc目标。

有些时候传入的投递地址可能会是本进程的地址，此时查找connection_resource的话会失败，因为当前并没有为本进程创建一个connection，从而导致投递消息失败，所以上层在投递消息的时候要专门为这个本地地址做过滤。这里的代码在发现dest是一个本进程地址之后，会将这个数据直接放到主循环消息队列中：

bool network_router::push_msg(std::shared_ptr<const std::string> from, std::shared_ptr<const std::string> dest, std::shared_ptr<const std::string> data, std::uint16_t cmd)
{
	if(dest->rfind(m_local_anchor_name, 0) == 0)
	{
		// 说明是本进程地址 直接推送数据到input_msg_queue
		network::con_msg_task local_msg_task;
		local_msg_task.first = {};
		local_msg_task.second = msg_task::construct(from, dest, data, cmd);
		m_input_msg_queue->push_msg(std::move(local_msg_task));
	}
	// 省略一些代码

}

找到connection之后的消息发送逻辑已经在之前的网络通信章节中介绍过了，读者可以回顾一下相关的内容来了解底层TCP传输细节，这里就不再介绍。当目标进程接收到了这个消息之后，主循环的on_msg回调首先解析出from,dest, msg, cmd这四个字段，然后根据cmd的类型与dest地址来做本进程的消息分发。由于具体的分发函数实现有点长，这里就只贴出space_server::on_msg中处理enums::packet_cmd::entity_raw_msg的部分：

case enums::packet_cmd::entity_raw_msg:
{
	if(!one_msg.dest)
	{
		m_logger->error("dest empty while handle rpc msg {}", *one_msg.data);
		return true;
	}
	
	utility::rpc_msg::call_result dispatch_result;

	auto real_dest = remove_local_anchor_prefix(*one_msg.dest);
	if(check_msg_forward(real_dest, one_msg))
	{
		return true;
	}
	dispatch_result = entity::entity_manager::instance().dispatch_entity_raw_msg(real_dest, cur_cmd_detail.second, one_msg.data);
	if (dispatch_result != utility::rpc_msg::call_result::suc)
	{
		m_logger->error("fail to dispatch raw_msg dest {} sync_cmd {} with error {}",  *one_msg.dest, cur_cmd_detail.second, int(dispatch_result));
	}
	return true;
}

这里的remove_local_anchor_prefix相当于把server_id::entity_id形式的dest解析出entity_id部分，check_msg_forward的函数行为我们后面再介绍，剩下的逻辑就是entity_manager根据entity_id去找到对应的entity并调用on_entity_raw_msg接口来处理本次数据:

utility::rpc_msg::call_result dispatch_entity_raw_msg(const std::string& dest, std::uint8_t cmd, std::shared_ptr<const std::string> msg)
{
	auto cur_entity = get_entity(dest);
	if (!cur_entity)
	{
		return utility::rpc_msg::call_result::dest_not_found;
	}
	return cur_entity->on_entity_raw_msg(cmd, msg);
}

这个on_entity_raw_msg再根据消息的子类型来做格式解析，并最终执行到rpc的分发函数on_rpc_msg:

utility::rpc_msg::call_result server_entity::on_entity_raw_msg(std::uint8_t cmd, std::shared_ptr<const std::string> msg_ptr)
{
	if(cmd == std::uint8_t(enums::entity_packet::json_rpc))
	{
		utility::rpc_msg e_msg;
		
		try
		{
			json msg = json::parse(*msg_ptr);
			msg.at("cmd").get_to(e_msg.cmd);
			msg.at("args").get_to(e_msg.args);
			auto from_iter = msg.find("from");
			if (from_iter != msg.end())
			{
				from_iter->get_to(e_msg.from);
			}
			auto err_iter = msg.find("err");
			if(err_iter != msg.end())
			{
				err_iter->get_to(e_msg.err);
			}
		}
		catch (std::exception& e)
		{
			m_logger->error("fail to decode rpc_msg {} error {}", *msg_ptr, e.what());
			return utility::rpc_msg::call_result::invalid_format;
		}
		return on_rpc_msg(e_msg);
	}
	
	return utility::rpc_msg::call_result::rpc_not_found;
}

on_rpc_msg之后的逻辑在rpc部分已经介绍过了，这里就不重复阐述。上面贴的代码对应的是space_server上的server_entity消息投递，其实与service_server上的base_service的消息投递机制基本没有差异，两者的on_rpc_msg机制是一样的，只不过一个被entity_manager中转分发，而另外一个被service_manager中转分发。类似的还有space_server上的manager_base，用来管理一些能够接受rpc的非server_entity单例数据:

utility::rpc_msg::call_result space_server::on_server_rpc_msg(const std::string& dest, const utility::rpc_msg& cur_rpc_msg)
{
	auto dispatch_result = entity::entity_manager::instance().dispatch_rpc_msg(dest, cur_rpc_msg);
	if(dispatch_result != utility::rpc_msg::call_result::dest_not_found)
	{
		return dispatch_result;
	}
	return manager_base::dispatch_rpc(dest, cur_rpc_msg);

}
void manager_base::init_managers(space_server* in_space_server)
{
	offline_msg_manager::instance().init(in_space_server);
	email_manager::instance().init(in_space_server);
	notify_manager::instance().init(in_space_server);
	rank_manager::instance().init(in_space_server);
	space_manager::instance().init(in_space_server);
}

不过service与server_entity在投递地址的表示上有很大的不同，server_entity的投递地址都是server_id::entity_id这个形式，而service的投递地址就只有service_type这个形式，没有绑定的server_id。这个解绑server_id的原因是为了支持service在不同的进程之间动态迁移从而实现负载均衡以及容灾，所以往特定service发消息的时候，只需要传递这个服务的名字即可作为投递地址：

void space_server::call_service( const std::string& service_name, const utility::rpc_msg& msg)
{
	auto dest_server = choose_server_for_service(service_name);
	if(dest_server.empty())
	{
		m_logger->warn("fail to find server for service {} msg {}", service_name,  json(msg).dump());
		return;
	}
	call_server(service_name, msg);
}

在一个service被创建的时候，每个进程都会接收到对应的创建通知，内部包含了所在的服务器信息，进程接收到这个消息之后会记录服务与进程之间的关联信息到m_services_by_id这个map成员上：

void space_server::on_notify_service_created(std::shared_ptr<network::net_connection> con, std::shared_ptr<const std::string> from, const json& msg)
{
	std::string service_type;
	std::string service_id;
	std::string service_server;
	try
	{
		msg.at("service_id").get_to(service_id);
		msg.at("service_type").get_to(service_type);
		msg.at("service_server").get_to(service_server);
	}
	catch(const std::exception& e)
	{
		m_logger->error("on_notify_service_created fail to parse msg {} error {}", msg.dump(), e.what());
		return;
	}
	m_services_by_id[service_id] = std::make_pair(service_type, service_server);
}

注意这里的key不是service_type而是一个根据规则生成的service唯一id, 这样做的理由是为了支持同一个service_type创建多个实例来做负载均衡。当需要给一个service发消息的时候，会查询是否有这个服务以及是否已经绑定了进程，这个绑定信息记录在m_services_by_pref上，如果没有绑定则遍历m_services_by_id里这个service_type对应的所有实例，随机选择其中的一个来执行绑定：

std::string space_server::choose_server_for_service(const std::string& service_name)
{
	auto temp_iter_1 = m_services_by_pref.find(service_name);
	if(temp_iter_1 != m_services_by_pref.end())
	{
		return temp_iter_1->second.second;
	}
	std::vector<std::pair<std::string, std::string>> temp_services;
	temp_services.reserve(8);
	for(const auto& one_pair: m_services_by_id)
	{
		if(one_pair.second.first != service_name)
		{
			continue;
		}
		temp_services.push_back(std::make_pair(one_pair.first, one_pair.second.second));
	}
	if(temp_services.empty())
	{
		return {};
	}
	auto cur_ms = utility::timer_manager::now_ts();
	auto cur_result = temp_services[cur_ms % temp_services.size()];
	m_services_by_pref[service_name] = cur_result;
	m_router->link_anchor_to_connection(service_name, cur_result.second);
	m_logger->info("set server {}  service_id {} to the pref server of service_type {}", cur_result.second, cur_result.first, service_name);
	return cur_result.second;

}

绑定之后就会在network_router中使用link_anchor_to_connection来注册投递地址与网络连接之间的映射，这里使用绑定而不是每次发送消息时都随机从temp_service中抽取一个的理由是保证从一个进程发送到一个投递地址的消息的接收顺序与发送顺序一致。

server_entity对于动态迁移的需求比service大很多，因为可迁移的server_entity数量相对service数量来说大好几个数量级。同时server_entity的迁移触发的比service更加频繁，每次切换场景时都可能触发，而service的迁移时机只有动态扩缩容和容灾。所以对于可迁移的server_entity的消息投递不能采取与service一样的广播推送最新地址的机制，这个机制会导致需要同步的信息量太多。所以针对server_entity的高频迁移特性，其在线消息投递使用的是一种基于中转的消息投递机制，对于这些可迁移的server_entity，在创建的时候就会在所在进程创建一个专门为其做消息转发服务的relay_entity:

player_entity* account_entity::create_player_entity(const std::string& player_id, const json& player_doc)
{
	std::string cur_err;
	auto cur_relay_entity_id = std::to_string(get_server()->gen_unique_uint64());
	json::object_t relay_init_info;
	relay_init_info["dest_eid"] = player_id;
	relay_init_info["dest_game"] = get_server()->local_server_info().name;
	auto cur_relay_entity = get_server()->create_entity("relay_entity", cur_relay_entity_id, gen_online_entity_id(),relay_init_info, cur_err);
	if(!cur_relay_entity)
	{
		m_logger->error("fail to create relay_entity");
		return nullptr;
	}
	m_relay_entity = dynamic_cast<relay_entity*>(cur_relay_entity);
	json::object_t player_init_info;
	player_init_info["prop"] = player_doc;
	player_init_info["prop"]["base"]["account_anchor"] = *get_call_proxy();
	player_init_info["prop"]["base"]["gate_name"] = m_gate_name;
	player_init_info["is_ghost"] = false;
	player_init_info["space_id"] = std::string();
	player_init_info["call_proxy"] = *cur_relay_entity->get_call_proxy();
	auto cur_entity = get_server()->create_entity("player_entity", player_id, gen_online_entity_id(),player_init_info, cur_err);
}

在创建好了relay_entity之后再创建所需的server_entity，同时将relay_entity的消息投递地址绑定在server_entity的call_proxy上。由于relay_entity是不可迁移的，所以使用call_proxy发送消息可以保证单进程有序与这个relay_entity进行消息通信。这个relay_entity会记录对应的server_entity的最新进程地址，server_entity每次迁移之前都需要通知relay_entity进行进程地址清空，等到relay_entity返回了迁移确认之后这个server_entity才能开始真正的迁移。在server_entity迁移成功之后，再将最新绑定的进程信息发送给relay_entity进行重新绑定。

void relay_entity::request_migrate_begin(const utility::rpc_msg& msg, const std::string& game_id, const std::string& space_id, const std::string& union_space_id, const json::object_t& enter_info)
{
	if(!m_dest_actor)
	{
		m_logger->error("request_migrate_begin while dest_anchor empty dest_game {} dest_eid {}", m_dest_game, m_dest_eid);
		return;
	}
	utility::rpc_msg reply_msg;
	reply_msg.cmd = "reply_migrate_begin";
	reply_msg.args.push_back(game_id);
	reply_msg.args.push_back(space_id);
	reply_msg.args.push_back(union_space_id);
	reply_msg.args.push_back(enter_info);
	call_server(m_dest_actor, reply_msg);
	m_dest_actor.reset();
	m_dest_game = game_id;
	
}

void relay_entity::notify_migrate_finish(const utility::rpc_msg& msg, const std::string& game_id)
{
	if(m_dest_game != game_id)
	{
		m_logger->error("notify_migrate_finish while  game not match  empty dest_game {} dest_eid {} new_game_id {}", m_dest_game, m_dest_eid, game_id);
		return;
	}
	m_dest_actor = std::make_shared<std::string>(utility::rpc_anchor::concat(m_dest_game, m_dest_eid));
	if(m_dest_game == get_local_server_name())
	{
		auto dest_entity = entity_manager::instance().get_entity(m_dest_eid);
		if(dest_entity)
		{
			for(const auto& one_msg:m_cached_msgs)
			{
				auto cur_cmd_detail = enums::packet_cmd_helper::decode(one_msg.cmd);
				auto dispatch_result = dest_entity->on_entity_raw_msg(cur_cmd_detail.second, one_msg.data);
				if(dispatch_result != utility::rpc_msg::call_result::suc)
				{
					m_logger->error("entity {} fail to dispatch cmd {} data {} err {}", m_dest_eid, cur_cmd_detail.second, *one_msg.data, std::uint8_t(dispatch_result));
				}
			}
			m_cached_msgs.clear();
		}
	}
	for(const auto& one_msg: m_cached_msgs)
	{
		auto cur_cmd_detail = enums::packet_cmd_helper::decode(one_msg.cmd);
		call_server(m_dest_actor, one_msg.data, cur_cmd_detail.first, cur_cmd_detail.second);
	}
	
	m_cached_msgs.clear();
}

当relay_entity接收到一个转发请求之后，先检查对应server_entity的进程绑定信息，如果没有绑定则先放到m_cached_msgs队列中，否则直接向对应的绑定进程发消息：

void relay_entity::forward_to_player(const network::msg_task& cur_msg_task)
{
	if(!m_dest_actor)
	{
		m_cached_msgs.push_back(cur_msg_task);
	}
	else
	{
		auto cur_cmd_detail = enums::packet_cmd_helper::decode(cur_msg_task.cmd);
		if(m_dest_game == get_local_server_name())
		{
			auto dest_entity = entity_manager::instance().get_entity(m_dest_eid);
			if(dest_entity)
			{
				auto dispatch_result = dest_entity->on_entity_raw_msg(cur_cmd_detail.second, cur_msg_task.data);
				if(dispatch_result != utility::rpc_msg::call_result::suc)
				{
					m_logger->error("fail to foward dispatch cmd {} data {} with result {}", cur_cmd_detail.second, *cur_msg_task.data, std::uint8_t(dispatch_result));
				}
			}
			else
			{
				m_logger->error("fail to find dest entity {}", m_dest_eid);
			}
		}
		else
		{
			call_server(m_dest_actor, cur_msg_task.data, cur_cmd_detail.first, cur_cmd_detail.second);
		}
		
	}
}

当重新绑定之后，还需要将m_cached_msgs全都按照接收顺序发送一遍。这样通过一个不可迁移的relay_entity我们就可以实现对可迁移server_entity的稳定有序消息投递了。

上面的relay_entity机制有一个小问题，就是消息发送方需要知道目标server_entity的call_proxy，如果无法获得这个call_proxy，则无法直接使用relay_entity机制。因此这里对于player_entity还有一个另外的在线通知机制，全局有一个login_service会记录所有在线玩家的account_entity地址，而account_entity也是一个不迁移的server_entity，可以保证其与relay_entity在同一个进程上。所以login_service上提供了一个request_call_online接口来执行向指定的player_id发送一个在线消息：

void login_service::request_call_online(const utility::rpc_msg& msg, const std::string& cur_player_id, const std::string& cmd, const std::vector<json>& args)
{
	auto cur_iter = m_online_players.find(cur_player_id);
	if(cur_iter == m_online_players.end())
	{
		m_logger->info("request_call_online not online fail to call {} cmd {} args {}", cur_player_id, cmd, json(args).dump());
		return;
	}
	utility::rpc_msg result_msg;
	result_msg.cmd = cmd;
	result_msg.args = args;
	auto cur_server = get_server();
	cur_server->call_server(cur_iter->second, result_msg);
}

当account_entity接收到这个消息之后，会调用call_player来手动获取这个同进程的relay_entity并执行转发：

void account_entity::call_player(const utility::rpc_msg& msg)
{
	if(!m_relay_entity)
	{
		return;
	}
	
	m_relay_entity->forward_to_player(msg);
}

这样在只拥有目标玩家的entity_id的情况下，可以通过login_service中转到account_entity，再中转到relay_entity，最终中转到player_entity。这样做会涉及到三次网络消息的收发，相对于基于call_proxy的中转来说多了一次，所以最好还是能够以call_proxy的方式去执行消息通知。因此很多service上都会有一个map记录当前所有在线玩家的call_proxy，玩家登录完成之后会向这些service推送自己的call_proxy，下面的就是玩家的聊天组件向聊天服务推送在线状态的代码，维护在线玩家call_proxy的相关代码：

void player_chat_component::on_login(bool is_relay)
{
	if(is_relay)
	{
		return;
	}
	utility::rpc_msg notify_msg;
	notify_msg.cmd = "notify_player_login";
	notify_msg.set_args(m_owner->entity_id(), *m_owner->get_call_proxy());
	m_owner->call_service("chat_service", notify_msg);
	
}

void player_chat_component::on_logout()
{
	utility::rpc_msg notify_msg;
	notify_msg.cmd = "notify_player_logout";
	notify_msg.set_args(m_owner->entity_id());
	m_owner->call_service("chat_service", notify_msg);
	
}

在聊天服务上使用一个unordered_map来记录注册过来的在线call_proxy, 同时提供一个封装好的call_online_player来处理向在线玩家发消息的需求:

void chat_service::notify_player_login(const utility::rpc_msg& msg, const std::string& player_id, const std::string& call_proxy)
{
	m_online_players[player_id] = std::make_shared<const std::string>(call_proxy);
}

void chat_service::notify_player_logout(const utility::rpc_msg& msg, const std::string& player_id)
{
	m_online_players.erase(player_id);
}
void chat_service::call_online_player(const std::string& player_id, const std::string& cmd, const std::vector<json>& args)
{
	auto cur_iter = m_online_players.find(player_id);
	if(cur_iter == m_online_players.end())
	{
		return;
	}
	utility::rpc_msg cur_msg;
	cur_msg.cmd = cmd;
	cur_msg.args = args;
	auto cur_server = get_server();
	cur_server->call_server(cur_iter->second, cur_msg);
}

就这样每个有通知在线玩家需求的service都自己维护了在线玩家的call_proxy记录，这样就可以避免都通过login_service去执行消息通知，降低login_service的单点压力。

多播消息

多播消息这种一般都是社群系统会使用到，例如群组、队伍、帮派等社群结构，多播系统的最简实现可以就只需要三行代码：使用一个for循环来遍历投递目标的集合然后调用单播消息投递接口。不过这样的实现会导致发送的消息rpc_msg被重复打包多次，浪费很多cpu，因此在server上提供了一个避免了重复消息打包的优化版本：

void space_server::call_server_multi(const entity::server_entity* cur_entity, const utility::rpc_msg& msg, const std::vector<std::string>& targets)
{
	auto shared_msg = msg.to_bytes();
	call_server_multi(cur_entity, shared_msg, enums::entity_packet::json_rpc, targets);
	
}

void space_server::call_server_multi(const entity::server_entity* cur_entity, std::shared_ptr<const std::string> msg, enums::entity_packet cur_entity_packet, const std::vector<std::string>& targets)
{
	for (auto one_dest : targets)
	{
		call_server(cur_entity, std::make_shared<std::string>(one_dest), msg, cur_entity_packet);
	}
}

这里使用rpc_msg上的to_bytes接口来预先执行消息打包序列化，生成一个shared_ptr<string>，这样就只需要打包一次，同时这个打包好的数据的生命周期能够被引用计数自动托管，业务层就不需要考虑msg生命周期的具体细节了。

不过这里的for循环也有一个可以性能优化的点，循环体内部会创建发送目标的一个shared_ptr<string>，其实更优的结果是创建一个包含了这个targets数组里所有元素的一个大的shared_ptr<string>，然后push_msg发送的时候使用下面的一个结构来编码dest:

struct shared_string_view
{
	std::shared_ptr<std::string> parent;
	std::string_view str;
};

这样make_shared只需要执行一次就行了，避免了多次动态内存分配相关的cpu损耗，同时整体的生命周期都被shared_ptr托管了，RAII会自动处理这个parent的资源释放。

有了这个call_server_multi接口之后，外部的多播接口只需要中转一下就好了：

void group_service::group_broadcast(const group_resource* cur_group, const std::string& cmd, const std::vector<json>& args, const std::string& except_id, bool without_leader)
{
	const auto& group_anchors = cur_group->get_online(except_id, without_leader);
	utility::rpc_msg cur_msg;
	cur_msg.cmd = cmd;
	cur_msg.args = args;
	get_server()->call_server_multi(cur_msg, group_anchors);
}

void team_service::team_broadcast(const team_resource* cur_team, const std::string& cmd, const std::vector<json>& args, const std::string& except_id)
{
	std::vector<std::string> team_anchors;
	team_anchors.reserve(cur_team->m_prop.m_players.index().size());
	for(const auto& one_idx: cur_team->m_prop.m_players.index())
	{
		auto cur_player_ptr = cur_team->m_prop.m_players.get_slot(one_idx.second);
		if(cur_player_ptr->m_id == except_id)
		{
			continue;
		}
		team_anchors.push_back(cur_player_ptr->anchor());
	}
	utility::rpc_msg cur_msg;
	cur_msg.cmd = cmd;
	cur_msg.args = args;
	get_server()->call_server_multi(cur_msg, team_anchors);
}

广播消息

游戏内有些逻辑需要往所有的在线玩家发送消息，这个需求虽然类似于多播，但是使用login_service去遍历所有玩家去调用多播接口的话消息流量会爆炸，而且会导致这个单点的卡顿，广播频率高的话可能会影响登录与下线。此时我们注意到这样的暴力广播流程里，很多时候数据的流向是login_service->game_server::relay_entity->game_server::player_entity->gate_server->client，中间的两层game_server不负责任何逻辑，完全执行数据转发任务。在这种情况下可以考虑跳过game_server相关的两层，直接把要广播的数据直接发向gate_server，然后让gate_server去遍历当前进程上绑定的所有客户端来执行在线通知。在这样的优化设计下，数据链路缩短为login_service->gate_server->client，减少了两层中转，同时数据流量也变得可控了，之前需要发送的数据份数为在线玩家的数量，而现在则缩减为gate_server的数量，这样内网流量放大倍率从几万的量级降低到了几十，同时还极大的降低了login_service的负载。

实际项目使用过程中，广播消息不仅仅只有全服广播，还有门派广播、势力广播、场景广播等多种类型。这些类型虽然不需要给所有在线客户端发送通知消息，但是其通知的人员范围依然是一个成百上千的数量，利用之前的多播接口依然会有性能问题。所以对于这些广播操作，采取先发送给所有的gate_server然后再由gate_server进行过滤后广播的方式依然可以节省很多的内网流量，同样可以避免这个广播接口的负载。在考虑了这些自定义的广播组需求之后，我们需要在gate_server上增加一个unordered_map来记录每个广播组下面的人员：

// 每个群组对应的client
// 这里的key string_view 对应的str就是value里的shared_ptr 
std::unordered_map<std::string, std::unordered_map<std::string_view, std::shared_ptr<const std::string>>> m_clients_for_group;

玩家角色可以通过下面的两个接口来维护自己与group之间的关系：

void space_server::update_player_broadcast_group(const entity::player_entity* cur_player, const std::vector<std::string>& groups)
{
	auto cur_account_id = *cur_player->shared_account_id();
	json params;
	params["entity_id"] = cur_account_id;
	params["groups"] = groups;
	json msg;
	msg["cmd"] = "request_link_group";
	msg["param"] = params;
	m_router->push_msg(cur_player->get_gate_id(), m_local_name_ptr, {}, std::make_shared<std::string>(msg.dump()), enums::packet_cmd_helper::encode(enums::packet_cmd::server_control, 0));
}

void space_server::clear_player_broadcast_group(const entity::player_entity* cur_player, const std::vector<std::string>& groups)
{
	auto cur_account_id = *cur_player->shared_account_id();
	json params;
	params["entity_id"] = cur_account_id;
	params["groups"] = groups;
	json msg;
	msg["cmd"] = "request_unlink_group";
	msg["param"] = params;
	m_router->push_msg(cur_player->get_gate_id(), m_local_name_ptr, {}, std::make_shared<std::string>(msg.dump()), enums::packet_cmd_helper::encode(enums::packet_cmd::server_control, 0));
}

gate_server上提供这两个接口来处理上面的两条消息通知，维护内部的m_clients_for_group:

void on_request_link_group(std::shared_ptr<network::net_connection> con, std::shared_ptr<const std::string> dest, const json& msg);

void on_request_unlink_group(std::shared_ptr<network::net_connection> con, std::shared_ptr<const std::string> dest, const json& msg);

同时gate_server上暴露一个广播接口on_request_broadcast_groups，方便space_server来调用，这个接口还支持给多个group一起广播:

void gate_server::on_request_broadcast_groups(std::shared_ptr<network::net_connection> con, std::shared_ptr<const std::string> dest, const json& msg)
{
	std::vector<std::string> groups;
	std::string msg_detail;
	std::uint8_t msg_cmd;
	try
	{
		msg.at("groups").get_to(groups);
		msg.at("msg_detail").get_to(msg_detail);
		msg.at("msg_cmd").get_to(msg_cmd);
	}
	catch(const std::exception& e)
	{
		m_logger->error("on_request_broadcast_groups fail to parse {} with error {}", msg.dump(), e.what());
		return;
	}
	auto cur_shared_rpc_msg = std::make_shared<std::string>(std::move(msg_detail));
	for(const auto& group: groups)
	{
		auto temp_iter = m_clients_for_group.find(group);
		if(temp_iter == m_clients_for_group.end())
		{
			continue;;
		}
		for(const auto& one_entity_pair: temp_iter->second)
		{
			m_router->push_msg({}, one_entity_pair.second, cur_shared_rpc_msg, enums::packet_cmd_helper::encode(enums::packet_cmd::game_to_client, msg_cmd));
		}
	}
	
}

space_server上提供了广播消息的入口:

void space_server::send_msg_to_broadcast_groups(const std::vector<std::string>& groups, std::shared_ptr<const std::string>& rpc_msg)
{
	json params;
	params["groups"] = groups;
	params["msg_detail"] = *rpc_msg;
	params["msg_cmd"] = std::uint8_t(enums::entity_packet::json_rpc);
	json msg;
	msg["cmd"] = "request_broadcast_groups";
	msg["param"] = params;
	auto cur_shared_msg = std::make_shared<std::string>(msg.dump());
	for(const auto& one_gate_info: m_gate_entities)
	{
		m_router->push_msg(one_gate_info.first, m_local_name_ptr, {}, std::make_shared<std::string>(msg.dump()), enums::packet_cmd_helper::encode(enums::packet_cmd::server_control, 0));
	}

}

而调用这个接口的目前只有专门处理广播的notify_manager:

void notify_manager::notify_broadcast_groups(const std::vector<std::string>& groups, const utility::rpc_msg& msg)
{
	auto cur_cmd = msg.cmd;
	std::shared_ptr<const std::string> shared_msg = msg.to_bytes();
	utility::rpc_msg stat_msg;
	stat_msg.cmd = "add_broadcast_group_stat";
	stat_msg.set_args(groups, cur_cmd, std::uint32_t(shared_msg->size()));
	m_server->call_service("notify_service", stat_msg);
	m_server->send_msg_to_broadcast_groups(groups, shared_msg);

}

目前的mosaic_game里只有全服聊天和门派聊天使用了notify_manager提供的这个接口来做广播: